From e7af91cc50fb699a54d38d790ca302bb1eb81881 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 10:35:22 -0400 Subject: [PATCH 01/33] plan: initial spine draft from Phase 1 sweep Seed the bench-v4 CI emitter dual-write spine from the parallel exploration sweep: goal, architecture decisions, out-of-scope, risks, scoped reviewer BANS, and accepted tradeoffs. Phase Map decomposition deferred to Step 1.4. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 148 ++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100644 .big-plans/ct__bench-v4-emitters.md diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md new file mode 100644 index 00000000000..290d7399766 --- /dev/null +++ b/.big-plans/ct__bench-v4-emitters.md @@ -0,0 +1,148 @@ +# bench-v4 CI emitter dual-write — big-plans spine + + +**Design spec:** `.big-plans/ct__bench-v4-emitters-design.md` (written by brainstorming in Step 1.2) +**Work shape:** feature-integration + +## Goal + +Make the Vortex monorepo CI emitters write benchmark results LIVE to the v4 RDS Postgres +("Path B / v4 dual-write") so the manual `vortex-bench-migrate` refresh is no longer needed, +shipping the write as an additive, best-effort, env-gated step that can never break the live +v2/v3 paths. + +## Architecture & key decisions + + + +- **Architecture:** port the already-written, unmerged v4 emitter from `origin/ct/bench-v4` + (commits `9a870091e` + `9a1824afa`, tree at tip `f9b36ae3f`) onto current `develop`; the + feature is FINISH + MERGE + PROVISION, not write-from-scratch. +- **Decision: one code PR (D) bracketed by external-infra ops phases (A before, C+B after).** + D produces a reviewable monorepo diff (SDD + gauntlet); A/B/C are CLI ops with side effects + and no in-repo diff (exit criteria are CLI verification commands, not code review). +- **Decision: the v4 step is dormant until the switch is flipped.** Every v4 workflow step is + gated on `vars.GH_BENCH_INGEST_ROLE_ARN != ''` with `continue-on-error: true`; merging D + with the var unset ships dead-but-safe code. Setting the var (phase B) is the live cutover. +- **Decision: `measurement_id` is computed client-side by a Python xxhash64 port** that must + reproduce the Rust reference (`benchmarks-website/server/src/db.rs`) byte-for-byte, validated + against the ported `scripts/measurement_id_golden.json` golden vectors. The cross-language + golden vectors ARE the contract. +- **Decision: the v3 `--server` path stays stdlib-only and intact;** the new v4 `--postgres` + path may use third-party deps (psycopg / IAM-token minting) but only via lazy import guarded + inside the postgres branch + declared in `post-ingest.py`'s PEP-723 block, so importing the + module or running `--server` never requires those deps. +- **Decision: recommended sequence (runbook §4):** A (create role) -> D (merge, dormant) -> + C (align token, gated go-ahead, redeploy v4) -> B (set ARN var + repoint URLs, flips ON) -> + soak/acceptance. Do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists and D is merged. +- **Decision: `SCHEMA_VERSION` stays in lockstep at `1`** across `post-ingest.py` and the v4 + schema; no bump in this project. + +## Out of scope + +- No changes to the live v2 (static S3 JSON) or v3 (DuckDB + `/api/ingest`) write paths beyond + leaving them fully intact; the v3 `--server` ingest step is preserved verbatim. +- No code lands in `vortex-data/benchmarks-website` — that repo (wire contract, golden vectors, + IAM provisioner, revalidate endpoint, RDS schema/migrations) is READ-ONLY reference. +- No RDS schema migrations or `migrate-schema.py` in the monorepo — schema/role management is + the extracted website repo's job (`schema-deploy.yml` + `GitHubBenchmarkSchemaRole`); the + monorepo emitter only CONNECTS as the already-provisioned `bench_ingest` role. +- Making v4 primary, retiring v3, DNS cutover, v2 decommission — all explicitly later, not here. +- No changes to `vortex-bench/src/v3.rs` record shapes (the wire records are unchanged). + +## Risks + +1. **Live-cutover blast radius (phase B)**: P=med; impact=moderate; mitigation: best-effort + + `continue-on-error` + env-gate means a v4 failure cannot fail a workflow; B is reversible by + unsetting `GH_BENCH_INGEST_ROLE_ARN`; watch the first emitting run before walking away. +2. **measurement_id port drift from the Rust reference**: P=med; impact=severe (silent wrong + upsert keys -> duplicate/again rows); mitigation: the ported pytest asserts every golden + vector byte-for-byte and is wired as a required CI check; covers Unicode + float + i32 edges. +3. **Revalidate 503/401 after cutover**: P=med; impact=minor (stale site, no data loss); cause: + Vercel prod missing `BENCH_REVALIDATE_TOKEN` (503) or token mismatch (401); mitigation: phase + C aligns one fresh token on both sides + redeploys before B flips the switch. +4. **AWS IAM-write reach unproven**: P=low; impact=moderate (phase A blocked); mitigation: verify + at phase A start; `bench-prod` is an IAM user with sibling role-create precedent. +5. **Vercel local copy not linked**: P=high; impact=minor (phase C friction); mitigation: run + `vercel link --scope vortex-data` (or pass `VERCEL_PROJECT_ID`) before any env op in C. +6. **Workflow YAML re-anchor onto diverged develop (`ci.yml`)**: P=low; impact=minor; mitigation: + the v4 inserts target intact anchors; `yamllint --strict` gates every workflow edit. + +--- + +## Current Position + +```yaml +phase: null +sub_phase: null +task: null +status: planning +last_gate: null +phase_entry_sha: null +``` + +--- + +## Phase Map + + + +(decomposition pending — Step 1.4) + +--- + +## Reviewer context + +### Project-specific BANS — constraints gauntlet reviewers MUST ENFORCE + + + +- **commits**: do NOT omit the `Signed-off-by: Connor Tsui ` DCO trailer on + any commit — DCO is enforced repo-wide. (Source: root `CLAUDE.md` § Commits.) +- **commit messages**: do NOT include a `---` scissors line or backticks in a `git commit -m` + body — the DCO pre-push hook false-positives on `---`, and backticks run as command + substitution. Use `-F`/heredoc and drop scissors. (Source: project memory.) +- **`.github/`**: do NOT land workflow YAML that fails `yamllint --strict -c .yamllint.yaml` + (double-quote when quoting, 1-space `{ }`, 0-space `[ ]`, 2-space inline comments, trailing + newline, no trailing spaces). (Source: `.github/AGENTS.md`.) +- **actions**: do NOT add an unpinned or tag-pinned `uses:` — pin to a full commit SHA with a + `# vN` comment, matching existing steps. (Source: existing workflows.) +- **Python**: do NOT introduce ruff `F,E,W,UP,I` failures or exceed line-length 120; new + `scripts/*.py` are linted by the repo-wide `ruff format`/`ruff check`. (Source: `pyproject.toml`.) +- **SPDX**: do NOT add a new `scripts/*.py` or `.github/` file without the two SPDX header lines. + (Source: `reuse.yml` + every existing file.) +- **v3 path protection**: do NOT add `continue-on-error` to (or otherwise weaken) the EXISTING + v3 `--server` ingest step, and do NOT make the v3 `--server` path or a bare module import + require third-party packages. (The NEW v4 step is intentionally `continue-on-error`; the NEW + v4 path may lazily import third-party deps — those are design, not violations.) +- **secrets**: do NOT echo, print, or pass a secret/token as a CLI arg or log-visible value; + secrets flow via `env:` and are read from the environment. (Source: existing workflows.) +- **SCHEMA_VERSION**: do NOT change `SCHEMA_VERSION` away from `1` — it must stay in lockstep + across the emitter and the v4 schema. (Source: `CONTRACT.md`.) +- **comments**: do NOT use em dashes in comments/docs (use `--`); full-sentence comments, own + lines, ~100-col. (Source: project memory + root `CLAUDE.md`.) + +### Carry-forward (DO NOT re-flag) + +#### Accepted tradeoffs + +- **v4 emitter third-party deps**: the v4 `--postgres` path depends on third-party packages + (psycopg + IAM-token minting) declared in `post-ingest.py`'s PEP-723 block and lazily imported + inside the postgres branch. Accepted: the v3 `--server` path stays stdlib-only and unaffected. +- **v4 step best-effort by design**: every v4 workflow step is `continue-on-error: true` and + env-gated. Accepted: a v4 failure intentionally does not fail the workflow (additive write). + +#### Deferred work + +- (none yet) + +--- + +## Verdict / Completion Ledger + + + +(none yet — execution has not started) From 3fa2fcb624a42e81880804b54a57548a1a8924e4 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 10:55:58 -0400 Subject: [PATCH 02/33] plan: spine updated with brainstorming design output Write the design spec and fold the three resolved design decisions into the spine: code-port scope (everything incl. extras), the 4-phase A/D/C/B structure with the ops-phase gauntlet/PR exemption, and the port-deps-as-is approach. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters-design.md | 242 +++++++++++++++++++++ .big-plans/ct__bench-v4-emitters.md | 14 +- 2 files changed, 253 insertions(+), 3 deletions(-) create mode 100644 .big-plans/ct__bench-v4-emitters-design.md diff --git a/.big-plans/ct__bench-v4-emitters-design.md b/.big-plans/ct__bench-v4-emitters-design.md new file mode 100644 index 00000000000..7e9705653d0 --- /dev/null +++ b/.big-plans/ct__bench-v4-emitters-design.md @@ -0,0 +1,242 @@ +# bench-v4 CI emitter dual-write — design spec + +**Status:** approved design (Phase 1 of big-plans) +**Work shape:** feature-integration (port + merge + provision) +**Spine:** `.big-plans/ct__bench-v4-emitters.md` +**Authoritative external plan:** `benchmarks-website/docs/runbooks/emitter-ingest-cutover.md` (verified 2026-06-19) +**Wire contract:** `benchmarks-website/CONTRACT.md` (SCHEMA_VERSION = 1) + +## 1. Goal & premise + +Make the Vortex monorepo CI emitters write benchmark results LIVE to the v4 RDS Postgres +("Path B / v4 dual-write") so the manual `vortex-bench-migrate` refresh is no longer needed. +The write is additive, best-effort (env-gated + `continue-on-error`), and cannot break the live +v2 (static S3 JSON) or v3 (DuckDB + `POST /api/ingest`) paths. + +**Key premise — most code already exists, unmerged.** The v4 emitter was implemented on +`origin/ct/bench-v4` (commits `9a870091e` Phase 1 + `9a1824afa` Phase 2; tree at tip +`f9b36ae3f`) and, crucially, was authored against the CURRENT top-level `scripts/` layout. So +this is FINISH + MERGE + PROVISION, not write-from-scratch. `scripts/post-ingest.py` has an +identical blob at the merge-base (`8acef3aab`) and current `develop` tip — mainline has not +touched it in the 89 intervening commits — so the +934/-22 Postgres-writer diff applies cleanly. +The five follow-up commits after `9a1824afa` are website-repo Phases 3-5 and do NOT touch the +emitter, so no follow-up folding is needed for the emitter artifacts. + +## 2. Architecture: dual-write, two paths from one emitter + +Both paths originate from the same `vortex-bench --gh-json-v3` JSONL output (bare records, no +`measurement_id` on the wire). `scripts/post-ingest.py` dispatches on its mode: + +- **Path A (v3, unchanged, hard-required):** `post-ingest.py --server $V3_INGEST_URL` wraps the + records in a `{run_meta, commit, records}` envelope and POSTs to `{server}/api/ingest` with + `Authorization: Bearer $INGEST_BEARER_TOKEN`. Stdlib-only (`urllib`/`json`/`subprocess`). + **Preserved verbatim.** +- **Path B (v4, new, best-effort):** `post-ingest.py --postgres $DSN --region $REGION` connects + to RDS as `bench_ingest` (RDS IAM auth, `sslmode=verify-full`), computes `measurement_id` + client-side, and runs `INSERT ... ON CONFLICT (measurement_id) DO UPDATE` upserts in one + transaction, then pings `POST {BENCH_SITE_BASE_URL}/api/revalidate` with + `Authorization: Bearer $BENCH_REVALIDATE_TOKEN`. The record shapes are identical to Path A. + +The v4 write is dormant until phase B sets `GH_BENCH_INGEST_ROLE_ARN`; every v4 workflow step is +gated on `vars.GH_BENCH_INGEST_ROLE_ARN != ''` with `continue-on-error: true`. + +### 2.1 The `measurement_id` contract (load-bearing) + +`scripts/_measurement_id.py` is a byte-for-byte Python port of the Rust reference in +`benchmarks-website/server/src/db.rs`. xxhash64, seed 0: + +- `hasher_for(tag)`: seed 0, write `tag` bytes, then `write_u8(0)` (per-table tag separator). +- `write_str(s)`: write `len(s) as u64`, then the UTF-8 bytes (length-prefixed). +- `write_opt_str(o)`: `Some` -> `write_u8(1)` + `write_str`; `None` -> `write_u8(0)`. +- `write_i32(v)`: `hasher.write_i32(v)`. +- `write_f64(v)`: `hasher.write_u64(v.to_bits())`. +- `finish`: `hasher.finish() as i64` (signed bit-cast — BIGINT is signed). + +Per-table field order (the byte layout the port must match exactly): + +| Table | Tag | Fields in order | +|---|---|---| +| `query_measurements` | `query_measurements` | commit_sha, dataset, dataset_variant(opt), scale_factor(opt), query_idx(i32), storage, engine, format | +| `compression_times` | `compression_times` | commit_sha, dataset, dataset_variant(opt), format, op | +| `compression_sizes` | `compression_sizes` | commit_sha, dataset, dataset_variant(opt), format | +| `random_access_times` | `random_access_times` | commit_sha, dataset, format | +| `vector_search_runs` | `vector_search_runs` | commit_sha, dataset, layout, flavor, threshold(f64) — `iterations` excluded | + +Validated by `scripts/test_measurement_id.py` against the ported `scripts/measurement_id_golden.json` +(825-line cross-language golden vectors covering empty strings, Unicode, `query_idx` i32 bounds, +and float edge cases). The golden vectors ARE the contract — the test is wired as a required CI +check. The two ported files' docstrings reference `benchmarks-website/server/src/db.rs` and +`.../tests/measurement_id_golden.rs`, which no longer live in the monorepo — repoint those doc +references at the extracted `vortex-data/benchmarks-website` repo. + +### 2.2 The v4 upsert (column lists, from `server/src/ingest.rs` + `migrations/001_initial_schema.sql`) + +Commit dim first (`ON CONFLICT (commit_sha) DO UPDATE`), then 5 fact upserts keyed on +`measurement_id`: + +- `query_measurements`: measurement_id, commit_sha, dataset, dataset_variant, scale_factor, query_idx, storage, engine, format, value_ns, all_runtimes_ns, peak_physical, peak_virtual, physical_delta, virtual_delta, env_triple +- `compression_times`: measurement_id, commit_sha, dataset, dataset_variant, format, op, value_ns, all_runtimes_ns, env_triple +- `compression_sizes`: measurement_id, commit_sha, dataset, dataset_variant, format, value_bytes +- `random_access_times`: measurement_id, commit_sha, dataset, format, value_ns, all_runtimes_ns, env_triple +- `vector_search_runs`: measurement_id, commit_sha, dataset, layout, flavor, threshold, value_ns, all_runtimes_ns, matches, rows_scanned, bytes_scanned, iterations, env_triple + +Non-finite f64 (NaN/Inf) must be rejected loud, never written. + +### 2.3 Third-party dependency handling (decided: port branch approach as-is) + +The v4 `--postgres` path needs psycopg + RDS IAM token minting. These are declared in +`post-ingest.py`'s PEP-723 inline `dependencies = [...]` block and **lazily imported only inside +the `--postgres` branch**. The v3 `--server` path and a bare `import post_ingest` stay +stdlib-only and dep-free. Scripts run via `uv run` (which respects PEP-723), matching the repo's +existing `uv run --no-project scripts/.py` convention. + +## 3. Code-port scope — phase D (decided: everything incl. extras) + +**IN (lands in `vortex-data/vortex`, ported from `f9b36ae3f`):** + +- `scripts/post-ingest.py` — add `--postgres`/`--region` mode (IAM-auth upsert, one txn, + NaN/Inf guard, `verify-full` TLS + post-connect `ssl_in_use` check, `bench_ingest` role + enforcement, best-effort `refresh_site_cache`); keep the v3 `--server` path intact. +- `scripts/_measurement_id.py` (new) — the xxhash64 port (repoint docstring). +- `scripts/measurement_id_golden.json` (new) — golden vectors (required, the test fails without it). +- `scripts/test_measurement_id.py` (new) — golden-vector pytest (repoint docstring). +- `scripts/test_post_ingest_postgres.py` (new) — testcontainers Postgres writer/upsert tests. +- `scripts/cross_check_python_writer.py` (new) — extra cross-check utility (later-phase extra). +- `scripts/test_post_ingest_revalidate.py` (new) — revalidate-ping tests (later-phase extra). +- `.github/workflows/bench.yml` — insert best-effort v4 step block after the v3 ingest step. +- `.github/workflows/sql-benchmarks.yml` — same, additionally gated `inputs.mode == 'develop'`. +- `.github/workflows/v3-commit-metadata.yml` — same (empty.jsonl, commit-row upsert); ADD + `id-token: write` permission (currently only `contents: read`). +- `.github/workflows/ci.yml` — wire the measurement_id pytest into the existing `python-test` + job (no docker), and add a docker-gated `scripts-test` job for the testcontainer tests. +- `pyproject.toml` / `uv.lock` — regenerate `uv.lock` with `uv lock` (do NOT copy the branch + blob; develop's lock diverged). + +**OUT (the extracted `vortex-data/benchmarks-website` repo owns these; do not land in monorepo):** + +- `migrations/*.sql`, `scripts/migrate-schema.py`, `scripts/test_migrate_schema.py`, + `.github/workflows/schema-deploy.yml` — schema/role management is the website repo's job + (`GitHubBenchmarkSchemaRole`); the monorepo emitter only CONNECTS as `bench_ingest`. +- All of `benchmarks-website/{server,migrate,web,infra}/**` — extracted repo. + +### 3.1 Workflow v4 step shape (per runbook §2.D) + +Each v4 step block: `aws-actions/configure-aws-credentials` assuming +`vars.GH_BENCH_INGEST_ROLE_ARN` (SHA-pinned action), download the RDS CA bundle, run +`post-ingest.py --postgres "$DSN" --region "$RDS_BENCH_REGION"`, then the revalidate ping to +`${BENCH_SITE_BASE_URL}/api/revalidate`. DSN: +`postgresql://bench_ingest@${RDS_BENCH_INSTANCE_ENDPOINT}:5432/${RDS_BENCH_DB_NAME}?sslmode=verify-full&sslrootcert=`. +Every v4 step: `if: vars.GH_BENCH_INGEST_ROLE_ARN != ''` + `continue-on-error: true`. `bench.yml` +and `sql-benchmarks.yml` already have `id-token: write` + a `configure-aws-credentials` step, so +the v4 step reuses that machinery (with the NEW ingest role ARN, distinct from the S3-scoped +`GitHubBenchmarkRole`). All edits must pass `yamllint --strict -c .yamllint.yaml`. + +## 4. big-plans phase structure (decided: 4 phases, A/D/C/B+soak) + +One code phase (D) bracketed by external-infra ops phases. **Ops phases (A, C, B) produce NO +in-repo diff** — they are direct CLI operations with side effects. Therefore ops phases: + +- do NOT run through `subagent-driven-development`; +- are NOT gauntlet-reviewed and do NOT open a PR (there is no diff to review or merge); +- have **machine-checkable CLI exit criteria** (role exists / var set / token returns 200); +- use a **pre-action external-side-effect confirmation** as the human checkpoint before each + mutating CLI op, and a post-action verification confirmation at the phase boundary. + +This is the sanctioned big-plans ops-phase adaptation (it is not "skipping rigor" — the rigor +is the CLI verification + the confirmation gate; gauntlet has no artifact to review). The code +phase (D) runs the full standard machinery (SDD per sub-phase + gauntlet pr-2/3 + phase-end +gauntlet + PR + human gate + squash-merge). + +**Phase order (runbook §4 safe ordering):** + +1. **Phase A (ops): provision the ingest IAM role.** Create `GitHubBenchmarkIngestRole` via + `benchmarks-website/infra/provision.sh` `ensure_ingest_role()` (idempotent) or a surgical + create. Trust `repo:vortex-data/vortex` (refs per the script); grant `rds-db:connect` as + `bench_ingest` on the instance `DbiResourceId`. Record the ARN. +2. **Phase D (code): port the v4 dual-write emitter + workflows.** The one reviewable PR; merges + with `GH_BENCH_INGEST_ROLE_ARN` unset, so the v4 code ships dormant-but-safe. +3. **Phase C (ops, GATED): align the revalidate token.** Generate one fresh token, set on the + v4 Vercel project (Production) AND the monorepo secret, redeploy v4 prod. Requires explicit + user go-ahead (was deferred in a prior session by an auto-mode guard). +4. **Phase B (ops, GATED — live cutover): flip the switch + repoint URLs + soak.** Set + `GH_BENCH_INGEST_ROLE_ARN` (the gate) and repoint `BENCH_SITE_BASE_URL` + + `BENCHMARKS_WEB_PROD_URL` to `https://benchmarks-website.vercel.app`; then await an emitting + run and verify acceptance (§6). Requires explicit user go-ahead. Reversible by unsetting the + var. + +Do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists AND D is merged. + +## 5. Reference values (verified 2026-06-19) + +- **AWS:** account `245040174862`, region `us-east-1`, profile `bench-prod` (IAM user + `connor-aws-cli`). GitHub OIDC provider `token.actions.githubusercontent.com` exists. +- **RDS:** endpoint `vortex-bench-prod.c4f8qygk4xdp.us-east-1.rds.amazonaws.com:5432`, database + `vortex_bench`. CA bundle `https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem`. + `DbiResourceId` via `aws rds describe-db-instances --db-instance-identifier vortex-bench-prod + --query 'DBInstances[0].DbiResourceId' --output text --profile bench-prod`. +- **IAM roles:** `GitHubBenchmarkIngestRole` (ABSENT, phase A creates it; ARN will be + `arn:aws:iam::245040174862:role/GitHubBenchmarkIngestRole`); `GitHubBenchmarkSchemaRole` + (exists). **DB roles:** `migrator` (DDL), `bench_ingest` (the v4 dual-write, `rds_iam`), + `bench_read` (Vercel reader, password). +- **Monorepo (`vortex-data/vortex`) GitHub config:** already-correct vars + `RDS_BENCH_DB_NAME=vortex_bench`, `RDS_BENCH_INSTANCE_ENDPOINT=vortex-bench-prod.c4f8qygk4xdp.us-east-1.rds.amazonaws.com`, + `RDS_BENCH_REGION=us-east-1`, + `GH_BENCH_SCHEMA_ROLE_ARN=arn:aws:iam::245040174862:role/GitHubBenchmarkSchemaRole`. To set in + phase B: `GH_BENCH_INGEST_ROLE_ARN` (absent now); repoint `BENCH_SITE_BASE_URL` + + `BENCHMARKS_WEB_PROD_URL` from the deleted `https://benchmarks-web.vercel.app` to + `https://benchmarks-website.vercel.app`. Secret `BENCH_REVALIDATE_TOKEN` exists (set + 2026-06-16) but is re-aligned in phase C. +- **Vercel:** project `benchmarks-website`, team `vortex-data`, orgId + `team_TkGBm7OlQtmqOFNpVNuaNpFX`, projectId `prj_AOss3j7VcSu5UoyBA1LIvj4G0DQ6`, + `https://benchmarks-website.vercel.app`, `develop` = production. Production env is MISSING + `BENCH_REVALIDATE_TOKEN` (phase C adds it). Local copy NOT linked — run + `vercel link --scope vortex-data` (or pass `VERCEL_PROJECT_ID`) before any env op. + +### 5.1 Phase B/C command reference (from runbook §2) + +Phase B: +``` +gh variable set GH_BENCH_INGEST_ROLE_ARN -R vortex-data/vortex --body 'arn:aws:iam::245040174862:role/GitHubBenchmarkIngestRole' +gh variable set BENCH_SITE_BASE_URL -R vortex-data/vortex --body 'https://benchmarks-website.vercel.app' +gh variable set BENCHMARKS_WEB_PROD_URL -R vortex-data/vortex --body 'https://benchmarks-website.vercel.app' +``` +Phase C (one fresh token on both sides, then redeploy v4 prod): +``` +TOKEN="$(python3 -c 'import secrets;print(secrets.token_urlsafe(48))')" +printf '%s' "$TOKEN" | vercel env add BENCH_REVALIDATE_TOKEN production +gh secret set BENCH_REVALIDATE_TOKEN -R vortex-data/vortex --body "$TOKEN" +``` +A new Vercel env var only takes effect on the next production deploy. + +## 6. Acceptance criteria (runbook §5) + +After phase B, trigger/await an emitting workflow and confirm: + +- the v4 step logs OIDC assume-role OK; +- the upsert reports `inserted`/`updated`; +- the revalidate ping returns `200 {revalidated:true}` (not 503/401 — 503 = Vercel prod missing + `BENCH_REVALIDATE_TOKEN` or not redeployed since; 401 = token mismatch); +- `curl -s https://benchmarks-website.vercel.app/api/health` shows `row_counts.commits` and + `latest_commit_timestamp` advancing to the just-run commit; +- the site shows the new commit with NO manual migration; +- the v3/v2 paths are unaffected (the workflow's v3 ingest step still succeeds). + +## 7. Risks + +1. **Live-cutover blast radius (B)**: P=med; mitigation: best-effort + `continue-on-error` + + env-gate; reversible by unsetting the var; watch the first run. +2. **measurement_id port drift**: P=med, impact=severe; mitigation: golden-vector test wired as + a required CI check, covering Unicode/float/i32 edges. +3. **Revalidate 503/401**: P=med, impact=minor; mitigation: phase C aligns one fresh token both + sides + redeploys before B. +4. **AWS IAM-write reach unproven**: P=low; verify at phase A start. +5. **Vercel local copy not linked**: P=high, impact=minor; run `vercel link` before phase C env op. +6. **Workflow YAML re-anchor onto diverged develop**: P=low; `yamllint --strict` gates each edit. + +## 8. Out of scope + +- Making v4 primary, retiring v3, DNS cutover, v2 decommission — all later. +- Any code in `vortex-data/benchmarks-website` (read-only reference). +- RDS schema migrations / role management in the monorepo (website repo owns them). +- Changes to `vortex-bench/src/v3.rs` record shapes or the v2/v3 write paths. +- SCHEMA_VERSION bump (stays 1). diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 290d7399766..0fde2869eff 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -20,9 +20,17 @@ v2/v3 paths. - **Architecture:** port the already-written, unmerged v4 emitter from `origin/ct/bench-v4` (commits `9a870091e` + `9a1824afa`, tree at tip `f9b36ae3f`) onto current `develop`; the feature is FINISH + MERGE + PROVISION, not write-from-scratch. -- **Decision: one code PR (D) bracketed by external-infra ops phases (A before, C+B after).** - D produces a reviewable monorepo diff (SDD + gauntlet); A/B/C are CLI ops with side effects - and no in-repo diff (exit criteria are CLI verification commands, not code review). +- **Decision: 4-phase structure** — Phase A (ops: create IAM role) -> Phase D (code: port + emitter + workflows, full SDD + gauntlet + PR + merge, dormant) -> Phase C (ops, GATED: align + revalidate token) -> Phase B (ops, GATED live cutover: set ARN var + repoint URLs + soak). + Ops phases A/C/B are CLI ops with side effects and NO in-repo diff: they skip gauntlet + PR; + exit criteria are CLI verification commands; the human checkpoint is a pre-action + external-side-effect confirmation (see design spec § 4 for the ops-phase adaptation rationale). +- **Decision: code-port scope = everything incl. extras** — the essentials (post-ingest.py + `--postgres`, `_measurement_id.py` + golden.json + test, the 3 workflow v4 steps, ci.yml + wiring) PLUS the testcontainer writer tests (`test_post_ingest_postgres.py`) PLUS the extras + (`cross_check_python_writer.py`, `test_post_ingest_revalidate.py`). Migrations + migrate-schema + are OUT (extracted website repo owns schema/roles). See design spec § 3. - **Decision: the v4 step is dormant until the switch is flipped.** Every v4 workflow step is gated on `vars.GH_BENCH_INGEST_ROLE_ARN != ''` with `continue-on-error: true`; merging D with the var unset ships dead-but-safe code. Setting the var (phase B) is the live cutover. From 865bd0029030f0b6e1d00cdc2a01f6943341d100 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:07:33 -0400 Subject: [PATCH 03/33] plan: spine updated after grill-me stress-test Fold grill-me outcomes into the spine and design spec: reorder phases to D -> A -> C -> B; add the demo-safety model (no prod RDS writes until phase B, phases C/B gated post-demo, pre-merge confidence via testcontainer PG16 + golden tests with no prod/develop dependency, real-RDS verified read-only at B); correct the deps mechanism to uv run --with (not PEP-723), the xxhash package usage, and the global CA bundle; record the verified v4 failure-isolation property and the testcontainer-vs-RDS gap risk. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters-design.md | 77 ++++++++++++++++------ .big-plans/ct__bench-v4-emitters.md | 40 +++++++---- 2 files changed, 85 insertions(+), 32 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters-design.md b/.big-plans/ct__bench-v4-emitters-design.md index 7e9705653d0..5cef06693a1 100644 --- a/.big-plans/ct__bench-v4-emitters-design.md +++ b/.big-plans/ct__bench-v4-emitters-design.md @@ -43,7 +43,9 @@ gated on `vars.GH_BENCH_INGEST_ROLE_ARN != ''` with `continue-on-error: true`. ### 2.1 The `measurement_id` contract (load-bearing) `scripts/_measurement_id.py` is a byte-for-byte Python port of the Rust reference in -`benchmarks-website/server/src/db.rs`. xxhash64, seed 0: +`benchmarks-website/server/src/db.rs`, using the `xxhash` package's `XXH64` (seed 0) — NOT a +hand-rolled hash. The length-prefix / opt / i32 / f64 framing below is what the port must +reproduce around that hash: - `hasher_for(tag)`: seed 0, write `tag` bytes, then `write_u8(0)` (per-table tag separator). - `write_str(s)`: write `len(s) as u64`, then the UTF-8 bytes (length-prefixed). @@ -84,11 +86,15 @@ Non-finite f64 (NaN/Inf) must be rejected loud, never written. ### 2.3 Third-party dependency handling (decided: port branch approach as-is) -The v4 `--postgres` path needs psycopg + RDS IAM token minting. These are declared in -`post-ingest.py`'s PEP-723 inline `dependencies = [...]` block and **lazily imported only inside -the `--postgres` branch**. The v3 `--server` path and a bare `import post_ingest` stay -stdlib-only and dep-free. Scripts run via `uv run` (which respects PEP-723), matching the repo's -existing `uv run --no-project scripts/.py` convention. +The v4 `--postgres` path needs `psycopg[binary]`, `boto3` (RDS IAM token minting), and `xxhash` +(the `measurement_id` hash). **Mechanism (corrected from grill-me — the code disagrees with the +runbook):** the deps are supplied at the call site via +`uv run --no-project --with 'psycopg[binary]' --with boto3 --with xxhash scripts/post-ingest.py`, +NOT declared in `post-ingest.py`'s PEP-723 block (which stays `dependencies = []`). They are +**lazily imported only inside the `--postgres` branch**, so the v3 `--server` path and a bare +`import post_ingest` stay stdlib-only and dep-free even though the deps are no longer pinned in +the file. (The test invocations and the ci.yml jobs must likewise provide `xxhash` / the +testcontainer deps via `--with` or the workspace.) ## 3. Code-port scope — phase D (decided: everything incl. extras) @@ -131,7 +137,29 @@ and `sql-benchmarks.yml` already have `id-token: write` + a `configure-aws-crede the v4 step reuses that machinery (with the NEW ingest role ARN, distinct from the S3-scoped `GitHubBenchmarkRole`). All edits must pass `yamllint --strict -c .yamllint.yaml`. -## 4. big-plans phase structure (decided: 4 phases, A/D/C/B+soak) +## 4. big-plans phase structure (decided: 4 phases, reordered D/A/C/B+soak) + +### 4.0 Demo-safety model (grill-me outcome — HARD CONSTRAINT) + +A live demo reads prod RDS in the hours after planning. Therefore: + +- **No prod RDS writes until phase B** (the live cutover), which is hard-gated to AFTER the demo + AND explicit user go-ahead. **Phase C also waits for after the demo** (it redeploys the v4 + site the demo may read). All external mutations (A, C, B) land post-demo; only the pure-code + phase D runs during the demo window (zero prod interaction). +- **Pre-merge confidence is built WITHOUT prod or develop** (the user's "be confident before we + merge" requirement): the **testcontainer Postgres tests** (`test_post_ingest_postgres.py`), + run against a local throwaway Postgres pinned to **RDS's major version (16)**, exercise the + full writer path — real upsert SQL, `measurement_id`, the single transaction, ON CONFLICT, + NaN/Inf guard — and the **golden-vector test** proves `measurement_id` byte-parity with Rust. + These are the load-bearing pre-merge gate (this is why "everything incl. extras" scope matters). +- The only thing local tests cannot prove — real-RDS IAM auth + `verify-full` TLS + role grant — + is verified post-demo at phase B via a **read-only / rolled-back** `bench_ingest` connect + (no data mutation), plus the first emitting run's `continue-on-error` logs. +- Reordered to **D -> A -> C -> B** (was A/D/C/B) so the next hours are spent only on code that + cannot affect the demo; every prod-touching op is deferred behind a post-demo gate. The + runbook's safety invariant still holds: do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists + AND D is merged — both precede phase B. One code phase (D) bracketed by external-infra ops phases. **Ops phases (A, C, B) produce NO in-repo diff** — they are direct CLI operations with side effects. Therefore ops phases: @@ -147,31 +175,36 @@ is the CLI verification + the confirmation gate; gauntlet has no artifact to rev phase (D) runs the full standard machinery (SDD per sub-phase + gauntlet pr-2/3 + phase-end gauntlet + PR + human gate + squash-merge). -**Phase order (runbook §4 safe ordering):** +**Phase order (reordered D -> A -> C -> B per §4.0):** -1. **Phase A (ops): provision the ingest IAM role.** Create `GitHubBenchmarkIngestRole` via +1. **Phase D (code): port the v4 dual-write emitter + workflows.** The one reviewable PR; merges + with `GH_BENCH_INGEST_ROLE_ARN` unset, so the v4 code ships dormant-but-safe. Pre-merge + confidence per §4.0 (testcontainer PG16 + golden). Zero prod interaction; demo-safe. +2. **Phase A (ops): provision the ingest IAM role.** Create `GitHubBenchmarkIngestRole` via `benchmarks-website/infra/provision.sh` `ensure_ingest_role()` (idempotent) or a surgical - create. Trust `repo:vortex-data/vortex` (refs per the script); grant `rds-db:connect` as - `bench_ingest` on the instance `DbiResourceId`. Record the ARN. -2. **Phase D (code): port the v4 dual-write emitter + workflows.** The one reviewable PR; merges - with `GH_BENCH_INGEST_ROLE_ARN` unset, so the v4 code ships dormant-but-safe. + create. Trust `repo:vortex-data/vortex` (refs per the script — develop is what the live + workflows run on); grant `rds-db:connect` as `bench_ingest` on the instance `DbiResourceId`. + Record the ARN. Data-safe (IAM only) but an external mutation -> post-demo, confirm first. 3. **Phase C (ops, GATED): align the revalidate token.** Generate one fresh token, set on the v4 Vercel project (Production) AND the monorepo secret, redeploy v4 prod. Requires explicit - user go-ahead (was deferred in a prior session by an auto-mode guard). -4. **Phase B (ops, GATED — live cutover): flip the switch + repoint URLs + soak.** Set + user go-ahead (was deferred in a prior session by an auto-mode guard). Post-demo (redeploy). +4. **Phase B (ops, GATED — live cutover): flip the switch + repoint URLs + soak.** First, the + read-only/rolled-back real-RDS `bench_ingest` verification (§4.0). Then set `GH_BENCH_INGEST_ROLE_ARN` (the gate) and repoint `BENCH_SITE_BASE_URL` + `BENCHMARKS_WEB_PROD_URL` to `https://benchmarks-website.vercel.app`; then await an emitting - run and verify acceptance (§6). Requires explicit user go-ahead. Reversible by unsetting the - var. + run and verify acceptance (§6). Requires explicit user go-ahead. Post-demo. Reversible by + unsetting the var. This is the FIRST prod RDS write of the project. -Do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists AND D is merged. +Do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists AND D is merged (both precede B by order). ## 5. Reference values (verified 2026-06-19) - **AWS:** account `245040174862`, region `us-east-1`, profile `bench-prod` (IAM user `connor-aws-cli`). GitHub OIDC provider `token.actions.githubusercontent.com` exists. - **RDS:** endpoint `vortex-bench-prod.c4f8qygk4xdp.us-east-1.rds.amazonaws.com:5432`, database - `vortex_bench`. CA bundle `https://truststore.pki.rds.amazonaws.com/us-east-1/us-east-1-bundle.pem`. + `vortex_bench`. CA bundle (what the workflow ships): the GLOBAL bundle + `https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem` (a superset; the runbook's + region-specific `us-east-1-bundle.pem` also works, but the code uses global). `DbiResourceId` via `aws rds describe-db-instances --db-instance-identifier vortex-bench-prod --query 'DBInstances[0].DbiResourceId' --output text --profile bench-prod`. - **IAM roles:** `GitHubBenchmarkIngestRole` (ABSENT, phase A creates it; ARN will be @@ -223,8 +256,14 @@ After phase B, trigger/await an emitting workflow and confirm: ## 7. Risks +0. **Demo-window prod-data corruption**: P=low-if-disciplined, impact=severe; the project must + not write/alter prod RDS during the demo. Mitigation: §4.0 — no prod RDS write before phase B; + C+B both hard-gated post-demo + go-ahead; only pure-code D runs in the demo window. 1. **Live-cutover blast radius (B)**: P=med; mitigation: best-effort + `continue-on-error` + env-gate; reversible by unsetting the var; watch the first run. +1a. **Testcontainer PG != RDS gap**: P=med, impact=moderate; local PG has no IAM auth / RDS TLS / + role grant, so pre-merge tests cannot prove that binding. Mitigation: pin testcontainer to PG + 16 (RDS major), and verify the real-RDS path read-only/rolled-back at phase B before the flip. 2. **measurement_id port drift**: P=med, impact=severe; mitigation: golden-vector test wired as a required CI check, covering Unicode/float/i32 edges. 3. **Revalidate 503/401**: P=med, impact=minor; mitigation: phase C aligns one fresh token both diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 0fde2869eff..dc8cf80b1b5 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -20,12 +20,18 @@ v2/v3 paths. - **Architecture:** port the already-written, unmerged v4 emitter from `origin/ct/bench-v4` (commits `9a870091e` + `9a1824afa`, tree at tip `f9b36ae3f`) onto current `develop`; the feature is FINISH + MERGE + PROVISION, not write-from-scratch. -- **Decision: 4-phase structure** — Phase A (ops: create IAM role) -> Phase D (code: port - emitter + workflows, full SDD + gauntlet + PR + merge, dormant) -> Phase C (ops, GATED: align - revalidate token) -> Phase B (ops, GATED live cutover: set ARN var + repoint URLs + soak). - Ops phases A/C/B are CLI ops with side effects and NO in-repo diff: they skip gauntlet + PR; - exit criteria are CLI verification commands; the human checkpoint is a pre-action - external-side-effect confirmation (see design spec § 4 for the ops-phase adaptation rationale). +- **Decision: 4-phase structure, reordered D -> A -> C -> B** (grill-me, demo-driven) — Phase D + (code: port emitter + workflows, full SDD + gauntlet + PR + merge, dormant) -> Phase A (ops: + create IAM role) -> Phase C (ops, GATED: align revalidate token) -> Phase B (ops, GATED live + cutover: set ARN var + repoint URLs + soak). Ops phases A/C/B are CLI ops with side effects and + NO in-repo diff: they skip gauntlet + PR; exit criteria are CLI verification commands; the + human checkpoint is a pre-action external-side-effect confirmation (see design spec § 4). +- **Decision: demo-safety model (HARD CONSTRAINT)** — NO prod RDS writes until phase B; phases C + and B both hard-gated to AFTER the demo + explicit go-ahead (C redeploys the v4 site the demo + reads). Only pure-code phase D runs in the demo window (zero prod interaction). Pre-merge + confidence comes from testcontainer Postgres tests (pinned to RDS major version 16) + the + golden-vector test, with NO prod/develop dependency; the real-RDS IAM/TLS path is verified + read-only/rolled-back at phase B before the flip. See design spec § 4.0. - **Decision: code-port scope = everything incl. extras** — the essentials (post-ingest.py `--postgres`, `_measurement_id.py` + golden.json + test, the 3 workflow v4 steps, ci.yml wiring) PLUS the testcontainer writer tests (`test_post_ingest_postgres.py`) PLUS the extras @@ -39,12 +45,15 @@ v2/v3 paths. against the ported `scripts/measurement_id_golden.json` golden vectors. The cross-language golden vectors ARE the contract. - **Decision: the v3 `--server` path stays stdlib-only and intact;** the new v4 `--postgres` - path may use third-party deps (psycopg / IAM-token minting) but only via lazy import guarded - inside the postgres branch + declared in `post-ingest.py`'s PEP-723 block, so importing the - module or running `--server` never requires those deps. -- **Decision: recommended sequence (runbook §4):** A (create role) -> D (merge, dormant) -> - C (align token, gated go-ahead, redeploy v4) -> B (set ARN var + repoint URLs, flips ON) -> - soak/acceptance. Do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists and D is merged. + path uses third-party deps (`psycopg[binary]`, `boto3`, `xxhash`) supplied at the call site via + `uv run --no-project --with ...` (NOT `post-ingest.py`'s PEP-723 block, which stays empty — + corrected from the runbook by grill-me), lazily imported only inside the postgres branch, so + importing the module or running `--server` never requires those deps. `_measurement_id.py` uses + the `xxhash` package's XXH64. CA bundle is the global bundle. +- **Decision: sequence (reordered):** D (build + merge dormant) -> A (create role) -> C (align + token, gated, post-demo, redeploy v4) -> B (set ARN var + repoint URLs, flips ON, post-demo) -> + soak/acceptance. Do NOT set `GH_BENCH_INGEST_ROLE_ARN` until A exists and D is merged (both + precede B by order). - **Decision: `SCHEMA_VERSION` stays in lockstep at `1`** across `post-ingest.py` and the v4 schema; no bump in this project. @@ -62,8 +71,13 @@ v2/v3 paths. ## Risks +0. **Demo-window prod-data corruption (next few hours)**: P=low-if-disciplined; impact=severe; + mitigation: no prod RDS write before phase B; phases C and B hard-gated post-demo + go-ahead; + only pure-code phase D runs during the demo window. Verified by failure isolation (the v4 + block is dormant until the ARN var is set, which is phase B). 1. **Live-cutover blast radius (phase B)**: P=med; impact=moderate; mitigation: best-effort + - `continue-on-error` + env-gate means a v4 failure cannot fail a workflow; B is reversible by + `continue-on-error` + env-gate means a v4 failure cannot fail a workflow (verified on-branch: + all v4 steps carry the gate + `continue-on-error`, v3 runs first); B is reversible by unsetting `GH_BENCH_INGEST_ROLE_ARN`; watch the first emitting run before walking away. 2. **measurement_id port drift from the Rust reference**: P=med; impact=severe (silent wrong upsert keys -> duplicate/again rows); mitigation: the ported pytest asserts every golden From 74a824257bda05a2be07ca6a7cdc53c14d7b47e4 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:12:14 -0400 Subject: [PATCH 04/33] plan: Phase Map decomposition Decompose the design into the Phase Map: Phase 1 (D, code) with sub-phases 1.1 measurement_id contract, 1.2 Postgres writer, 1.3 CI + workflow wiring (SDD + gauntlet + PR + human gate, phase-4 depth); Phases 2-4 (A/C/B, ops) as direct-CLI phases with machine-checkable CLI exit criteria and no gauntlet/PR. Add the Orchestration notes section documenting the ops-phase protocol and the dedicated-cleanup-PR wrap-up for resume. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 49 +++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index dc8cf80b1b5..455755dce56 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -109,9 +109,54 @@ phase_entry_sha: null ## Phase Map - + + +| Phase | Sub-phase | Scope (one line) | Exit criteria (command → expected) | Sub-phase gauntlet | Phase gauntlet | Task-plan pointer | +|---|---|---|---|---|---|---| +| 1: D — port v4 emitter (CODE) | 1.1 measurement_id contract | Port `_measurement_id.py` (xxhash XXH64) + `measurement_id_golden.json` + `test_measurement_id.py`; repoint the two docstrings off the extracted-repo paths | (phase-level — see exit row) | pr-2 | | `.big-plans/ct__bench-v4-emitters--1-1-measurement-id.plan.md` | +| *(phase 1 cont.)* | 1.2 Postgres writer | Add `--postgres`/`--region` mode to `post-ingest.py` (RDS IAM auth, `verify-full` TLS + `ssl_in_use` check, `bench_ingest` enforce, NaN/Inf guard, 5-table upsert + commit dim in one txn, best-effort revalidate) keeping `--server` stdlib-only intact; add `test_post_ingest_postgres.py` (testcontainers PG16), `test_post_ingest_revalidate.py`, `cross_check_python_writer.py` | | pr-3 | | `.big-plans/ct__bench-v4-emitters--1-2-postgres-writer.plan.md` | +| *(phase 1 cont.)* | 1.3 CI + workflow wiring | Wire the contract test into `ci.yml` `python-test` (docker-UNgated) + a docker-gated `scripts-test` job for the testcontainer tests; add the best-effort v4 step block to `bench.yml`, `sql-benchmarks.yml`, `v3-commit-metadata.yml` (+ `id-token: write` on the last); regenerate `uv.lock` with `uv lock` | | pr-3 | | `.big-plans/ct__bench-v4-emitters--1-3-ci-workflow-wiring.plan.md` | +| *(phase 1 exit)* | *(all sub-phases)* | v4 dual-write emitter + dormant workflow steps ported; v3 path intact | `uv run --no-project --with xxhash pytest scripts/test_measurement_id.py` → 0; `yamllint --strict -c .yamllint.yaml .github/workflows/bench.yml .github/workflows/sql-benchmarks.yml .github/workflows/v3-commit-metadata.yml .github/workflows/ci.yml` → 0; `python3 scripts/post-ingest.py --help` → 0 (stdlib-only); testcontainer writer tests green under docker | | phase-4 | | +| 2: A — provision IAM role (OPS) | 2.1 create ingest role | Create `GitHubBenchmarkIngestRole` (trust `repo:vortex-data/vortex` on develop; grant `rds-db:connect` as `bench_ingest` on the instance `DbiResourceId`) via `provision.sh` `ensure_ingest_role` or a surgical create; record the ARN. Pre-action confirm | (phase-level) | n/a (ops) | | n/a (ops — direct CLI) | +| *(phase 2 exit)* | *(ops)* | Ingest role exists with the rds-db:connect grant | `aws iam get-role --role-name GitHubBenchmarkIngestRole --profile bench-prod` → 0 AND `aws iam get-role-policy --role-name GitHubBenchmarkIngestRole --policy-name rds-db-connect-ingest --profile bench-prod` shows `rds-db:connect` for `bench_ingest` | | n/a (ops) | | +| 3: C — align revalidate token (OPS, GATED) | 3.1 set + align token | `vercel link --scope vortex-data`; generate one fresh token; `vercel env add BENCH_REVALIDATE_TOKEN production`; `gh secret set BENCH_REVALIDATE_TOKEN -R vortex-data/vortex`; redeploy v4 prod. GATED: explicit go-ahead + post-demo. Pre-action confirm | (phase-level) | n/a (ops) | | n/a (ops — direct CLI) | +| *(phase 3 exit)* | *(ops)* | Token aligned both sides + v4 redeployed | authed `POST {site}/api/revalidate` with the token → HTTP `200 {revalidated:true}` (not 503/401) | | n/a (ops) | | +| 4: B — flip switch + soak (OPS, GATED, live cutover) | 4.1 pre-flip verify + cutover | read-only/rolled-back `bench_ingest` RDS verify (IAM + `verify-full` TLS, no data write); then `gh variable set GH_BENCH_INGEST_ROLE_ARN` + repoint `BENCH_SITE_BASE_URL` + `BENCHMARKS_WEB_PROD_URL` to `https://benchmarks-website.vercel.app`. GATED: explicit go-ahead + post-demo. Pre-action confirm | (phase-level) | n/a (ops) | | n/a (ops — direct CLI) | +| *(phase 4 cont.)* | 4.2 soak + acceptance | trigger/await an emitting `develop` run; verify §5 acceptance (OIDC assume-role ok, upsert inserted/updated, revalidate 200, `/api/health` advances, v3 step still green) | | n/a (ops) | | n/a (ops — direct CLI) | +| *(phase 4 exit)* | *(ops)* | v4 dual-write live + acceptance green | `gh variable list -R vortex-data/vortex` shows `GH_BENCH_INGEST_ROLE_ARN` set + `BENCH_SITE_BASE_URL`=`https://benchmarks-website.vercel.app`; `curl -s https://benchmarks-website.vercel.app/api/health` `latest_commit_timestamp` advanced to a post-cutover commit; the emitting run's v3 step still succeeded | | n/a (ops) | | -(decomposition pending — Step 1.4) +--- + +## Orchestration notes (custom spine — READ ON RESUME) + +This spine mixes one CODE phase with three OPS phases; the orchestrator handles them differently. + +- **Phase 1 (D) — CODE — standard big-plans:** per sub-phase, `writing-plans` (JIT) -> + `subagent-driven-development` -> `gauntlet` (pr-2/pr-3) checkpoint; phase-end `gauntlet` + (phase-4); open the phase PR; mandatory human gate; squash-merge. Pre-merge confidence per + design spec § 4.0 (testcontainer PG16 + golden, no prod/develop dependency). +- **Phases 2-4 (A/C/B) — OPS — adapted:** NO `writing-plans`, NO `subagent-driven-development`, + NO `gauntlet`, NO PR (there is no in-repo diff to plan, review, or merge). Execute each + sub-phase as direct CLI. Before EACH mutating external op (AWS / GitHub / Vercel), fire the + pre-action external-side-effect confirmation. The phase exit criterion is the CLI verification + command in the Phase Map. The phase-boundary human-gate AUQ still fires, but the user reviews + the CLI verification output instead of a PR diff. Spine status for ops phases: + `implementing` while running the CLI ops (resume = re-run idempotently), then + `awaiting-human-gate` at the boundary (skip `reviewing`/`fixing` — no gauntlet). Do NOT route + an ops phase's `implementing` status into the SDD/writing-plans loop. +- **GATED ops phases (C and B):** in addition to the per-op confirmation, both require explicit + user go-ahead AND must wait until AFTER the demo (design spec § 4.0). C is the deferred + `BENCH_REVALIDATE_TOKEN` un-deferral; B is the live cutover (first prod RDS write). +- **Wrap-up (after phase B):** the spine + design spec rode onto develop via phase D's + squash-merge (big-plans' normal between-phase behavior; nothing sensitive — all infra values + already exist in the repo's workflows/vars). Because the trailing ops phases A/C/B have NO PR, + remove the scaffolding from develop with a dedicated `chore: remove big-plans spine` cleanup PR + rather than relying on a final phase PR. The branch-local A/C/B `plan:` commits are the + orchestration audit trail and are not merged. --- From a3ffeeea8ad1c9147b31a0a4ece5233143975f32 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:16:04 -0400 Subject: [PATCH 05/33] =?UTF-8?q?plan:=20transition=20to=20Phase=202=20?= =?UTF-8?q?=E2=80=94=20begin=20execution?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enter Phase 1 (D, code) at sub-phase 1.1 (measurement_id contract): status implementing, phase_entry_sha null (filled by the next commit). Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 455755dce56..756fcc9413c 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -97,10 +97,10 @@ v2/v3 paths. ## Current Position ```yaml -phase: null -sub_phase: null +phase: "1: D — port v4 emitter (CODE)" +sub_phase: "1.1 measurement_id contract" task: null -status: planning +status: implementing last_gate: null phase_entry_sha: null ``` From e5a0cafca14df93e364525bdeb6017aa3c6ca6a3 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:16:28 -0400 Subject: [PATCH 06/33] plan: record phase_entry_sha for phase 1 Fill phase_entry_sha with commit 1's SHA (closes the two-commit entry window). Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 756fcc9413c..2cb3353d530 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -102,7 +102,7 @@ sub_phase: "1.1 measurement_id contract" task: null status: implementing last_gate: null -phase_entry_sha: null +phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` --- From a84975a79c3e677b59a1483cfb262fa087535199 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:19:35 -0400 Subject: [PATCH 07/33] plan: task-plan for sub-phase 1.1 (measurement_id contract) JIT writing-plans output for sub-phase 1.1: extract the three measurement_id contract files from f9b36ae3f, repoint docstrings, verify golden parity. Signed-off-by: Connor Tsui --- ...ch-v4-emitters--1-1-measurement-id.plan.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 .big-plans/ct__bench-v4-emitters--1-1-measurement-id.plan.md diff --git a/.big-plans/ct__bench-v4-emitters--1-1-measurement-id.plan.md b/.big-plans/ct__bench-v4-emitters--1-1-measurement-id.plan.md new file mode 100644 index 00000000000..32e6e6e815b --- /dev/null +++ b/.big-plans/ct__bench-v4-emitters--1-1-measurement-id.plan.md @@ -0,0 +1,132 @@ +# Sub-phase 1.1 — measurement_id contract Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Port the three `measurement_id` contract files verbatim from branch ref `f9b36ae3f` into the monorepo working tree, repoint the cross-repo doc references, and verify byte-parity against the golden vectors. + +**Architecture:** This is an EXTRACT-FROM-BRANCH + ADJUST + VERIFY unit, not a from-scratch implementation. The files already exist on the unmerged branch and apply cleanly (mainline has never touched these paths). `scripts/_measurement_id.py` reproduces the Rust server's `measurement_id` xxhash64 hash; `scripts/test_measurement_id.py` pins it against `scripts/measurement_id_golden.json` (63 cross-language golden vectors over 5 fact tables). + +**Tech Stack:** Python 3.11+, the `xxhash` package (XXH64), pytest, `uv run --no-project --with ...`, ruff. + +## Global Constraints + +- Extract files VERBATIM from `f9b36ae3f` via `git show f9b36ae3f: > `. Do NOT re-author or alter any hash logic, field order, or golden value. +- Repoint ONLY the cross-repo documentation references in the two `.py` docstrings (paths that no longer exist in this monorepo). No logic changes. +- Both `.py` files MUST keep the two SPDX header lines: `# SPDX-License-Identifier: Apache-2.0` and `# SPDX-FileCopyrightText: Copyright the Vortex contributors` (already present on the branch). +- Python lint: ruff `F,E,W,UP,I` clean; line-length 120. Comments use `--`, never em dashes; full sentences. +- This sub-phase touches ONLY: `scripts/_measurement_id.py`, `scripts/measurement_id_golden.json`, `scripts/test_measurement_id.py`. Do NOT touch `ci.yml`, any workflow, or `post-ingest.py` (those are sub-phase 1.3 / 1.2). +- Commits: DCO trailer `Signed-off-by: Connor Tsui `. Never put backticks or a `---` scissors line in a `git commit -m` message; use `git commit -F` with a heredoc. + +--- + +### Task 1: Port the measurement_id contract files and verify golden parity + +**Files:** +- Create: `scripts/_measurement_id.py` (extracted from `f9b36ae3f`) +- Create: `scripts/measurement_id_golden.json` (extracted from `f9b36ae3f`) +- Create/Test: `scripts/test_measurement_id.py` (extracted from `f9b36ae3f`) + +**Interfaces:** +- Produces (relied on by sub-phases 1.2/1.3): `scripts/_measurement_id.py` exposing the dispatch dict `MEASUREMENT_ID_BY_TABLE` keyed by the five table names (`query_measurements`, `compression_times`, `compression_sizes`, `random_access_times`, `vector_search_runs`), each value a callable taking the table's dimensional fields as keyword args and returning a signed-i64 `measurement_id`. Sub-phase 1.2's Postgres writer imports this module to compute upsert keys. +- Consumes: nothing from earlier tasks (this is the first sub-phase). + +- [ ] **Step 1: Extract the golden vectors and the test from the branch (the "test" half first, for a real red->green)** + +```bash +cd "$(git rev-parse --show-toplevel)" +git show f9b36ae3f:scripts/measurement_id_golden.json > scripts/measurement_id_golden.json +git show f9b36ae3f:scripts/test_measurement_id.py > scripts/test_measurement_id.py +``` + +Do NOT edit `measurement_id_golden.json` at all (not even its `note` field): it is a Rust-generated artifact (regenerated by `REGEN_GOLDEN_VECTORS=1 cargo test -p vortex-bench-server --test measurement_id_golden` in the `vortex-data/benchmarks-website` repo). Editing it would drift the monorepo copy from the generator. Porting it byte-identical is what lets the test pin cross-language parity. + +- [ ] **Step 2: Run the test to verify it fails (the implementation module does not exist yet)** + +```bash +uv run --no-project --with pytest --with xxhash pytest scripts/test_measurement_id.py -q +``` + +Expected: an ERROR during collection — `scripts/test_measurement_id.py` executes `port = _load_port()` at import time, which tries to load `scripts/_measurement_id.py`; that file does not exist yet, so collection fails (FileNotFoundError / module load error). This confirms the test is wired to the implementation. + +- [ ] **Step 3: Extract the implementation module from the branch** + +```bash +git show f9b36ae3f:scripts/_measurement_id.py > scripts/_measurement_id.py +``` + +- [ ] **Step 4: Run the test to verify it passes (all 63 vectors + the 3 meta tests)** + +```bash +uv run --no-project --with pytest --with xxhash pytest scripts/test_measurement_id.py -q +``` + +Expected: PASS — 63 parametrized `test_python_port_matches_golden` cases plus `test_all_tables_covered` and `test_multibyte_fixture_present`, all green (66 passed). If any vector mismatches, the port and the golden file disagree — STOP and report; do not edit values to force a pass. + +- [ ] **Step 5: Repoint the cross-repo doc references in `_measurement_id.py` (cosmetic; no logic change)** + +In `scripts/_measurement_id.py`, replace the single occurrence of the path token: + +- old: `` `benchmarks-website/server/src/db.rs` `` (in the module docstring, the "byte-for-byte port of" line) +- new: ``the `vortex-data/benchmarks-website` repo's `server/src/db.rs` `` + +so the sentence reads "This is a byte-for-byte port of the `vortex-data/benchmarks-website` repo's `server/src/db.rs` (`measurement_id_*`, `hasher_for`, ...)". This is the only `benchmarks-website` reference in this file. + +- [ ] **Step 6: Repoint the cross-repo doc references in `test_measurement_id.py` (cosmetic; no logic change)** + +In `scripts/test_measurement_id.py` docstring, repoint both references: + +- `` `benchmarks-website/server/src/db.rs` `` -> ``the `vortex-data/benchmarks-website` repo's `server/src/db.rs` `` +- `` `benchmarks-website/server/tests/measurement_id_golden.rs` `` -> ``the `vortex-data/benchmarks-website` repo's `server/tests/measurement_id_golden.rs` `` + +Do NOT alter the `import`s, the loader, or any assertion. + +- [ ] **Step 7: Re-run the test after the docstring edits (guard against an accidental code edit)** + +```bash +uv run --no-project --with pytest --with xxhash pytest scripts/test_measurement_id.py -q +``` + +Expected: PASS (still 66 passed). The docstring edits must not change behavior. + +- [ ] **Step 8: Lint and format-check the two new Python files** + +```bash +uvx ruff check scripts/_measurement_id.py scripts/test_measurement_id.py +uvx ruff format --check scripts/_measurement_id.py scripts/test_measurement_id.py +``` + +Expected: both exit 0 (no diagnostics, already-formatted). If `ruff format --check` reports a diff, run `uvx ruff format scripts/_measurement_id.py scripts/test_measurement_id.py` and re-run Step 7 to confirm the test still passes. + +- [ ] **Step 9: Confirm SPDX headers are present on both `.py` files** + +```bash +head -2 scripts/_measurement_id.py +head -2 scripts/test_measurement_id.py +``` + +Expected: each begins with `# SPDX-License-Identifier: Apache-2.0` then `# SPDX-FileCopyrightText: Copyright the Vortex contributors`. (They are present on the branch; this is a guard.) + +- [ ] **Step 10: Commit** + +```bash +git add scripts/_measurement_id.py scripts/measurement_id_golden.json scripts/test_measurement_id.py +git commit -F - <<'EOF' +scripts: add measurement_id Python port + golden-vector test + +Port the server-internal measurement_id xxhash64 hash to Python +(scripts/_measurement_id.py) with its 63-vector cross-language golden test +(scripts/test_measurement_id.py, scripts/measurement_id_golden.json), extracted +verbatim from the v4 emitter branch. Repoint the docstrings to the extracted +vortex-data/benchmarks-website repo. The golden vectors pin Rust == Python. + +Signed-off-by: Connor Tsui +EOF +``` + +--- + +## Self-Review + +- **Spec coverage:** all three files (`_measurement_id.py`, `measurement_id_golden.json`, `test_measurement_id.py`) are created (Steps 1, 3); docstrings repointed (Steps 5-6); SPDX verified (Step 9); parity verified (Steps 4, 7); lint clean (Step 8). Out-of-scope items (ci.yml, workflows, post-ingest.py) are untouched. +- **Placeholder scan:** none — every step has an exact command or an exact old->new string. +- **Type consistency:** the `MEASUREMENT_ID_BY_TABLE` dispatch and the five table names match the golden file's `table` values and what sub-phase 1.2 will import. From ec31812ce5b797615c16fe903c5d65be198ef859 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:22:26 -0400 Subject: [PATCH 08/33] scripts: add measurement_id Python port + golden-vector test Port the server-internal measurement_id xxhash64 hash to Python (scripts/_measurement_id.py) with its 63-vector cross-language golden test (scripts/test_measurement_id.py, scripts/measurement_id_golden.json), extracted verbatim from the v4 emitter branch. Repoint the docstrings to the extracted vortex-data/benchmarks-website repo. The golden vectors pin Rust == Python. Signed-off-by: Connor Tsui --- scripts/_measurement_id.py | 220 ++++++++ scripts/measurement_id_golden.json | 825 +++++++++++++++++++++++++++++ scripts/test_measurement_id.py | 102 ++++ 3 files changed, 1147 insertions(+) create mode 100644 scripts/_measurement_id.py create mode 100644 scripts/measurement_id_golden.json create mode 100644 scripts/test_measurement_id.py diff --git a/scripts/_measurement_id.py b/scripts/_measurement_id.py new file mode 100644 index 00000000000..bea1858d01e --- /dev/null +++ b/scripts/_measurement_id.py @@ -0,0 +1,220 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +"""Python port of the server-internal `measurement_id` xxhash64 functions. + +This is a byte-for-byte port of the `vortex-data/benchmarks-website` repo's +`server/src/db.rs` (`measurement_id_*`, `hasher_for`, +`write_str`, `write_opt_str`, `write_i32`, `write_f64`, `finish`). The v4 ingest +writer (PR-2.1) computes the same `measurement_id` the v3 Rust server computes, +so re-ingesting an existing `(commit, dim-tuple)` upserts the existing row via +`ON CONFLICT (measurement_id) DO UPDATE` instead of inserting a duplicate. + +The equivalence is NOT assumed -- it is pinned by golden vectors generated from +the Rust source of truth (`scripts/measurement_id_golden.json`, regenerated by +the `measurement_id_golden_vectors` test in `db.rs`) and checked by +`scripts/test_measurement_id.py`. Any drift in either implementation fails that +test. + +## The hash, precisely + +`measurement_id` is a canonical xxhash64 (seed 0) over a byte buffer built as: + +1. The per-table tag bytes, then a single `0x00` separator (`hasher_for`). +2. For each dimensional field, in the exact order the Rust function writes it: + - `write_str(s)` -> `len(utf8(s))` as a little-endian u64, then the utf8 + bytes. The length is the BYTE length, not the character count. + - `write_opt_str(o)` -> `0x00` for `None`; `0x01` then `write_str(s)` for + `Some(s)`. + - `write_i32(v)` -> 4 little-endian two's-complement bytes. + - `write_f64(v)` -> the IEEE-754 bit pattern as 8 little-endian bytes + (`v.to_bits()` written little-endian == `struct.pack(" i64` (Postgres/DuckDB `BIGINT` is + signed), matching Rust's `hasher.finish() as i64`. + +Canonical xxhash64 is streaming-equivalent: feeding the bytes incrementally (as +the Rust `Hasher` does via `write_*`) yields the same digest as hashing the +fully concatenated buffer once (as this module does). + +The little-endian integer encodings are load-bearing and assume a little-endian +host. The Rust side serializes integers with native-endian byte order +(twox-hash 2.x `write_u64` / `write_i32` use `to_ne_bytes`, and `write_f64` is +`write_u64(v.to_bits())`), so byte-for-byte compatibility holds only where +`to_ne_bytes == to_le_bytes`. Every target in play (x86_64 / aarch64 CI runners, +dev machines, the RDS Postgres host, the Vercel reader) is little-endian, and the +golden vectors are generated on a little-endian host and pin it there. On a +big-endian host both this module and the Rust `write_*` would have to switch to a +shared explicit endianness. +""" + +import struct + +import xxhash + +# Must match `XxHash64::with_seed(0)` in `db.rs::hasher_for`. +_SEED = 0 + +# Per-table tag literals. These MUST match the `hasher_for("")` argument in +# each `db.rs::measurement_id_*` function verbatim; the tag is the table name. +_TAG_QUERY_MEASUREMENTS = "query_measurements" +_TAG_COMPRESSION_TIMES = "compression_times" +_TAG_COMPRESSION_SIZES = "compression_sizes" +_TAG_RANDOM_ACCESS_TIMES = "random_access_times" +_TAG_VECTOR_SEARCH_RUNS = "vector_search_runs" + + +def _hasher_buf(tag: str) -> bytearray: + """Start a hash buffer seeded with a per-table tag plus a `0x00` separator. + + Mirrors `db.rs::hasher_for`: two fact tables that share the same dim values + still produce distinct `measurement_id`s because the tag differs. + """ + buf = bytearray() + buf += tag.encode("utf-8") + buf.append(0) + return buf + + +def _write_str(buf: bytearray, s: str) -> None: + """Append a length-prefixed string: utf8 BYTE length as LE u64, then bytes.""" + encoded = s.encode("utf-8") + buf += struct.pack(" None: + """Append an optional string: `0x00` for None, `0x01` + the string for Some.""" + if s is None: + buf.append(0) + else: + buf.append(1) + _write_str(buf, s) + + +def _write_i32(buf: bytearray, v: int) -> None: + """Append a 32-bit signed integer as 4 little-endian two's-complement bytes.""" + buf += struct.pack(" None: + """Append a 64-bit float as its 8 little-endian IEEE-754 bytes. + + `struct.pack(" int: + """Hash the buffer (xxhash64, seed 0) and bit-cast the u64 digest to i64.""" + digest = xxhash.xxh64(bytes(buf), seed=_SEED).intdigest() + # Bit-cast u64 -> i64 to match Rust's `hasher.finish() as i64`. + return digest - (1 << 64) if digest >= (1 << 63) else digest + + +def measurement_id_query( + *, + commit_sha: str, + dataset: str, + dataset_variant: str | None, + scale_factor: str | None, + query_idx: int, + storage: str, + engine: str, + format: str, +) -> int: + """`measurement_id` for a `query_measurements` row. Mirrors + `db.rs::measurement_id_query`.""" + buf = _hasher_buf(_TAG_QUERY_MEASUREMENTS) + _write_str(buf, commit_sha) + _write_str(buf, dataset) + _write_opt_str(buf, dataset_variant) + _write_opt_str(buf, scale_factor) + _write_i32(buf, query_idx) + _write_str(buf, storage) + _write_str(buf, engine) + _write_str(buf, format) + return _finish(buf) + + +def measurement_id_compression_time( + *, + commit_sha: str, + dataset: str, + dataset_variant: str | None, + format: str, + op: str, +) -> int: + """`measurement_id` for a `compression_times` row. Mirrors + `db.rs::measurement_id_compression_time`.""" + buf = _hasher_buf(_TAG_COMPRESSION_TIMES) + _write_str(buf, commit_sha) + _write_str(buf, dataset) + _write_opt_str(buf, dataset_variant) + _write_str(buf, format) + _write_str(buf, op) + return _finish(buf) + + +def measurement_id_compression_size( + *, + commit_sha: str, + dataset: str, + dataset_variant: str | None, + format: str, +) -> int: + """`measurement_id` for a `compression_sizes` row. Mirrors + `db.rs::measurement_id_compression_size`.""" + buf = _hasher_buf(_TAG_COMPRESSION_SIZES) + _write_str(buf, commit_sha) + _write_str(buf, dataset) + _write_opt_str(buf, dataset_variant) + _write_str(buf, format) + return _finish(buf) + + +def measurement_id_random_access( + *, + commit_sha: str, + dataset: str, + format: str, +) -> int: + """`measurement_id` for a `random_access_times` row. Mirrors + `db.rs::measurement_id_random_access`. Note: no `dataset_variant`.""" + buf = _hasher_buf(_TAG_RANDOM_ACCESS_TIMES) + _write_str(buf, commit_sha) + _write_str(buf, dataset) + _write_str(buf, format) + return _finish(buf) + + +def measurement_id_vector_search( + *, + commit_sha: str, + dataset: str, + layout: str, + flavor: str, + threshold: float, +) -> int: + """`measurement_id` for a `vector_search_runs` row. Mirrors + `db.rs::measurement_id_vector_search`. `iterations` is intentionally NOT part + of the dim tuple -- it is a side count.""" + buf = _hasher_buf(_TAG_VECTOR_SEARCH_RUNS) + _write_str(buf, commit_sha) + _write_str(buf, dataset) + _write_str(buf, layout) + _write_str(buf, flavor) + _write_f64(buf, threshold) + return _finish(buf) + + +# Dispatch table keyed by the fact-table name, used by the golden-vector test to +# map a vector's `table` field to the matching port function. Keeping it here (vs. +# in the test) means a new fact table is wired in one place alongside the port. +MEASUREMENT_ID_BY_TABLE = { + _TAG_QUERY_MEASUREMENTS: measurement_id_query, + _TAG_COMPRESSION_TIMES: measurement_id_compression_time, + _TAG_COMPRESSION_SIZES: measurement_id_compression_size, + _TAG_RANDOM_ACCESS_TIMES: measurement_id_random_access, + _TAG_VECTOR_SEARCH_RUNS: measurement_id_vector_search, +} diff --git a/scripts/measurement_id_golden.json b/scripts/measurement_id_golden.json new file mode 100644 index 00000000000..1813a720d27 --- /dev/null +++ b/scripts/measurement_id_golden.json @@ -0,0 +1,825 @@ +{ + "note": "Golden vectors for the measurement_id_* hash, generated from benchmarks-website/server/src/db.rs (the source of truth). Regenerate with REGEN_GOLDEN_VECTORS=1 cargo test -p vortex-bench-server --test measurement_id_golden. scripts/test_measurement_id.py asserts the Python port reproduces every measurement_id here.", + "seed": 0, + "vectors": [ + { + "fields": { + "commit_sha": "0000000000000000000000000000000000000000", + "dataset": "dataset-0", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -20, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 8941066430020068021, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000009e3779b97f4a7c15", + "dataset": "dataset-1", + "dataset_variant": "variant-1", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -19, + "scale_factor": "sf-1", + "storage": "s3" + }, + "measurement_id": 3149995011396621799, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000003c6ef372fe94f82a", + "dataset": "dataset-2", + "dataset_variant": "variant-2", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -18, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -6215740072949592228, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000daa66d2c7ddf743f", + "dataset": "dataset-3", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -17, + "scale_factor": "sf-3", + "storage": "s3" + }, + "measurement_id": 8041805059478031959, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000078dde6e5fd29f054", + "dataset": "dataset-4", + "dataset_variant": "variant-4", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -16, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 4772600626526399345, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000001715609f7c746c69", + "dataset": "dataset-5", + "dataset_variant": "variant-5", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -15, + "scale_factor": "sf-5", + "storage": "s3" + }, + "measurement_id": -9016235375010445132, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000b54cda58fbbee87e", + "dataset": "dataset-6", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -14, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -6411293753624450917, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000538454127b096493", + "dataset": "dataset-7", + "dataset_variant": "variant-7", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -13, + "scale_factor": "sf-7", + "storage": "s3" + }, + "measurement_id": 8460757218662476151, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000f1bbcdcbfa53e0a8", + "dataset": "dataset-8", + "dataset_variant": "variant-8", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -12, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 7670946045315319280, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000008ff34785799e5cbd", + "dataset": "dataset-9", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -11, + "scale_factor": "sf-9", + "storage": "s3" + }, + "measurement_id": 1978441881557979666, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000002e2ac13ef8e8d8d2", + "dataset": "dataset-10", + "dataset_variant": "variant-10", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -10, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 8135912901467482164, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000cc623af8783354e7", + "dataset": "dataset-11", + "dataset_variant": "variant-11", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -9, + "scale_factor": "sf-11", + "storage": "s3" + }, + "measurement_id": 805733765350646006, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000006a99b4b1f77dd0fc", + "dataset": "dataset-12", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -8, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -4091026328830870798, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000008d12e6b76c84d11", + "dataset": "dataset-13", + "dataset_variant": "variant-13", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -7, + "scale_factor": "sf-13", + "storage": "s3" + }, + "measurement_id": -6061385115965648155, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000a708a824f612c926", + "dataset": "dataset-14", + "dataset_variant": "variant-14", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -6, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -6684883584448916863, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000454021de755d453b", + "dataset": "dataset-15", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -5, + "scale_factor": "sf-15", + "storage": "s3" + }, + "measurement_id": 8814896453978445645, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000e3779b97f4a7c150", + "dataset": "dataset-16", + "dataset_variant": "variant-16", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -4, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 6686567714742361044, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000081af155173f23d65", + "dataset": "dataset-17", + "dataset_variant": "variant-17", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -3, + "scale_factor": "sf-17", + "storage": "s3" + }, + "measurement_id": 3557917168552872470, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000001fe68f0af33cb97a", + "dataset": "dataset-18", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -2, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -6165762661706299592, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000be1e08c47287358f", + "dataset": "dataset-19", + "dataset_variant": "variant-19", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": -1, + "scale_factor": "sf-19", + "storage": "s3" + }, + "measurement_id": -535031789163192453, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000005c55827df1d1b1a4", + "dataset": "dataset-20", + "dataset_variant": "variant-20", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 0, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 4634990895302954655, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000fa8cfc37711c2db9", + "dataset": "dataset-21", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 1, + "scale_factor": "sf-21", + "storage": "s3" + }, + "measurement_id": 5936866269340399925, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000098c475f0f066a9ce", + "dataset": "dataset-22", + "dataset_variant": "variant-22", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 2, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 4471015533681449689, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000036fbefaa6fb125e3", + "dataset": "dataset-23", + "dataset_variant": "variant-23", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 3, + "scale_factor": "sf-23", + "storage": "s3" + }, + "measurement_id": 3270457410965634007, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000d5336963eefba1f8", + "dataset": "dataset-24", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 4, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -6630843892626754319, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000736ae31d6e461e0d", + "dataset": "dataset-25", + "dataset_variant": "variant-25", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 5, + "scale_factor": "sf-25", + "storage": "s3" + }, + "measurement_id": 5238548701197127815, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000011a25cd6ed909a22", + "dataset": "dataset-26", + "dataset_variant": "variant-26", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 6, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -869863853831808211, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000afd9d6906cdb1637", + "dataset": "dataset-27", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 7, + "scale_factor": "sf-27", + "storage": "s3" + }, + "measurement_id": -6175971040302506030, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000004e115049ec25924c", + "dataset": "dataset-28", + "dataset_variant": "variant-28", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 8, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 4758474844754348116, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000ec48ca036b700e61", + "dataset": "dataset-29", + "dataset_variant": "variant-29", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 9, + "scale_factor": "sf-29", + "storage": "s3" + }, + "measurement_id": 4027371045171197062, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000008a8043bceaba8a76", + "dataset": "dataset-30", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 10, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 6213625536713947544, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "00000000000000000000000028b7bd766a05068b", + "dataset": "dataset-31", + "dataset_variant": "variant-31", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 11, + "scale_factor": "sf-31", + "storage": "s3" + }, + "measurement_id": 3264730627889711995, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000c6ef372fe94f82a0", + "dataset": "dataset-32", + "dataset_variant": "variant-32", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 12, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 6426500585598625566, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000006526b0e96899feb5", + "dataset": "dataset-33", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 13, + "scale_factor": "sf-33", + "storage": "s3" + }, + "measurement_id": -470855441762601549, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000035e2aa2e7e47aca", + "dataset": "dataset-34", + "dataset_variant": "variant-34", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 14, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -8702944831802119265, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000a195a45c672ef6df", + "dataset": "dataset-35", + "dataset_variant": "variant-35", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 15, + "scale_factor": "sf-35", + "storage": "s3" + }, + "measurement_id": 6237767006190293538, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000003fcd1e15e67972f4", + "dataset": "dataset-36", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 16, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": 5322839569260259127, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "000000000000000000000000de0497cf65c3ef09", + "dataset": "dataset-37", + "dataset_variant": "variant-37", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 17, + "scale_factor": "sf-37", + "storage": "s3" + }, + "measurement_id": 1995931045924102080, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000007c3c1188e50e6b1e", + "dataset": "dataset-38", + "dataset_variant": "variant-38", + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 18, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -7932518466629682266, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0000000000000000000000001a738b426458e733", + "dataset": "dataset-39", + "dataset_variant": null, + "engine": "vortex", + "format": "vortex-file-compressed", + "query_idx": 19, + "scale_factor": "sf-39", + "storage": "s3" + }, + "measurement_id": 7227626056758470989, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "tpch", + "dataset_variant": null, + "engine": "vortex", + "format": "parquet", + "query_idx": 7, + "scale_factor": "1", + "storage": "nvme" + }, + "measurement_id": 6778969912115220139, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "데이터셋-日本語", + "dataset_variant": "café-Ω", + "engine": "ヴ", + "format": "format-✓", + "query_idx": 0, + "scale_factor": "σ", + "storage": "s3" + }, + "measurement_id": 8038081617766937373, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "boundary", + "dataset_variant": null, + "engine": "e", + "format": "f", + "query_idx": -2147483648, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -1997262439178828726, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "boundary", + "dataset_variant": null, + "engine": "e", + "format": "f", + "query_idx": 2147483647, + "scale_factor": null, + "storage": "nvme" + }, + "measurement_id": -1111737216055603595, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "", + "dataset_variant": "", + "engine": "", + "format": "", + "query_idx": 0, + "scale_factor": "", + "storage": "" + }, + "measurement_id": -4121125782605521286, + "table": "query_measurements" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "chimp", + "dataset_variant": null, + "format": "vortex", + "op": "encode" + }, + "measurement_id": -1351591591404870516, + "table": "compression_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "chimp", + "dataset_variant": null, + "format": "vortex", + "op": "decode" + }, + "measurement_id": 8221793816172450272, + "table": "compression_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "taxi", + "dataset_variant": "v2", + "format": "parquet", + "op": "encode" + }, + "measurement_id": -192124532873811017, + "table": "compression_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "압축-テスト", + "dataset_variant": "変種", + "format": "vortex-✓", + "op": "decode" + }, + "measurement_id": -2942165613664654219, + "table": "compression_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "chimp", + "dataset_variant": null, + "format": "vortex" + }, + "measurement_id": 4543757287978490921, + "table": "compression_sizes" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "taxi", + "dataset_variant": "v2", + "format": "parquet" + }, + "measurement_id": -7313339766868556874, + "table": "compression_sizes" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "크기-サイズ", + "dataset_variant": "変種-Ω", + "format": "vortex-✓" + }, + "measurement_id": -8576081510595521142, + "table": "compression_sizes" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "chimp", + "format": "vortex" + }, + "measurement_id": 4734678989716641189, + "table": "random_access_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "taxi", + "format": "parquet" + }, + "measurement_id": 164255076009922704, + "table": "random_access_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "無作為", + "format": "✓" + }, + "measurement_id": 832111591737631247, + "table": "random_access_times" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 0.0 + }, + "measurement_id": 2427216480405807826, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 0.5 + }, + "measurement_id": -1711405494360824574, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 0.83 + }, + "measurement_id": 2194658981253663824, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 1.0 + }, + "measurement_id": -5633397304139473003, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": -1.5 + }, + "measurement_id": -5505640980838817192, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 1e-9 + }, + "measurement_id": -812426564273547024, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 1000000000000.0 + }, + "measurement_id": 8044969450004818277, + "table": "vector_search_runs" + }, + { + "fields": { + "commit_sha": "0123456789abcdef0123456789abcdef01234567", + "dataset": "cohere-large-10m", + "flavor": "vortex", + "layout": "trained", + "threshold": 3.141592653589793 + }, + "measurement_id": 3687626387279485390, + "table": "vector_search_runs" + } + ] +} diff --git a/scripts/test_measurement_id.py b/scripts/test_measurement_id.py new file mode 100644 index 00000000000..ce3128f3354 --- /dev/null +++ b/scripts/test_measurement_id.py @@ -0,0 +1,102 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +"""Cross-language golden-vector test for the `measurement_id` Python port. + +`scripts/_measurement_id.py` ports the server-internal xxhash64 functions from +the `vortex-data/benchmarks-website` repo's `server/src/db.rs`. This test asserts +the port reproduces the golden vectors generated FROM that Rust source of truth +(`scripts/measurement_id_golden.json`, written by the +`measurement_id_golden_vectors` test in the `vortex-data/benchmarks-website` +repo's `server/tests/measurement_id_golden.rs`). + +Because the Rust test computes the golden ids with the real `measurement_id_*` +functions and asserts the committed file matches them, and this test asserts the +Python port matches the same committed file, the two implementations are pinned +transitively: Rust == golden == Python. Either side drifting fails its half. +""" + +import importlib.util +import json +from pathlib import Path + +import pytest + +_SCRIPTS_DIR = Path(__file__).resolve().parent +_GOLDEN_PATH = _SCRIPTS_DIR / "measurement_id_golden.json" + + +def _load_port(): + """Import `_measurement_id.py` by file path for symmetry with the sibling + `test_migrate_schema.py` loader; a normal import also works since the module + name is a valid identifier, but loading by path keeps the scripts/ tests + uniform and independent of `sys.path` setup.""" + spec = importlib.util.spec_from_file_location( + "_measurement_id", _SCRIPTS_DIR / "_measurement_id.py" + ) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +port = _load_port() + + +def _golden_vectors() -> list[dict]: + document = json.loads(_GOLDEN_PATH.read_text(encoding="utf-8")) + assert document["seed"] == 0, "port assumes xxhash64 seed 0" + vectors = document["vectors"] + assert vectors, "golden file has no vectors; regenerate via the Rust test" + return vectors + + +# Build the parametrize list once at import so each golden vector is its own test +# case: a single failure names the exact diverging vector (`NN-`). +_VECTORS = _golden_vectors() +_PARAMS = [pytest.param(v, id=f"{i:02d}-{v['table']}") for i, v in enumerate(_VECTORS)] + + +@pytest.mark.parametrize("vector", _PARAMS) +def test_python_port_matches_golden(vector: dict) -> None: + """The Python port reproduces the Rust-generated `measurement_id` for one + golden vector.""" + table = vector["table"] + fields = vector["fields"] + expected = vector["measurement_id"] + + func = port.MEASUREMENT_ID_BY_TABLE.get(table) + assert func is not None, f"no port function for table {table!r}" + + actual = func(**fields) + + assert actual == expected, ( + f"measurement_id mismatch for {table} vector {fields!r}: " + f"port produced {actual}, Rust golden is {expected}" + ) + # The id is bit-cast to i64; confirm it stays in range (catches a port that + # forgot the u64->i64 conversion and returned a bare u64). + assert -(2**63) <= actual < 2**63, "measurement_id must be a signed 64-bit int" + + +def test_all_tables_covered() -> None: + """Every fact table in the port's dispatch table appears in the golden + vectors, so no family silently lacks cross-language coverage.""" + golden_tables = {v["table"] for v in _VECTORS} + port_tables = set(port.MEASUREMENT_ID_BY_TABLE) + assert port_tables == golden_tables, ( + f"port tables {sorted(port_tables)} != golden tables {sorted(golden_tables)}; " + "a fact family is missing cross-language coverage" + ) + + +def test_multibyte_fixture_present() -> None: + """Guard the regression that motivated byte-length prefixing: at least one + vector must carry multibyte UTF-8, or the port could use char-length and + still pass. If the golden set is ever regenerated without a multibyte + fixture, fail loudly.""" + blob = json.dumps(_VECTORS, ensure_ascii=False) + assert any(ord(ch) > 0x7F for ch in blob), ( + "no multibyte UTF-8 in golden vectors; the byte-vs-char length-prefix " + "regression is unguarded" + ) From 766f58f0b05402186dca7cac9ade9a89d2a69cfe Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:25:16 -0400 Subject: [PATCH 09/33] scripts: ruff-format measurement_id test to repo line-length Collapse three lines the verbatim port carried from the source branch's narrower line-length so the file passes `ruff format --check` (repo uses 120). Whitespace-only; no test logic or values change. Signed-off-by: Connor Tsui --- scripts/test_measurement_id.py | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/scripts/test_measurement_id.py b/scripts/test_measurement_id.py index ce3128f3354..88a40202b86 100644 --- a/scripts/test_measurement_id.py +++ b/scripts/test_measurement_id.py @@ -31,9 +31,7 @@ def _load_port(): `test_migrate_schema.py` loader; a normal import also works since the module name is a valid identifier, but loading by path keeps the scripts/ tests uniform and independent of `sys.path` setup.""" - spec = importlib.util.spec_from_file_location( - "_measurement_id", _SCRIPTS_DIR / "_measurement_id.py" - ) + spec = importlib.util.spec_from_file_location("_measurement_id", _SCRIPTS_DIR / "_measurement_id.py") assert spec is not None and spec.loader is not None module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) @@ -71,8 +69,7 @@ def test_python_port_matches_golden(vector: dict) -> None: actual = func(**fields) assert actual == expected, ( - f"measurement_id mismatch for {table} vector {fields!r}: " - f"port produced {actual}, Rust golden is {expected}" + f"measurement_id mismatch for {table} vector {fields!r}: port produced {actual}, Rust golden is {expected}" ) # The id is bit-cast to i64; confirm it stays in range (catches a port that # forgot the u64->i64 conversion and returned a bare u64). @@ -97,6 +94,5 @@ def test_multibyte_fixture_present() -> None: fixture, fail loudly.""" blob = json.dumps(_VECTORS, ensure_ascii=False) assert any(ord(ch) > 0x7F for ch in blob), ( - "no multibyte UTF-8 in golden vectors; the byte-vs-char length-prefix " - "regression is unguarded" + "no multibyte UTF-8 in golden vectors; the byte-vs-char length-prefix regression is unguarded" ) From 21457221764e28f705323984a9dead280e55ddba Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:28:30 -0400 Subject: [PATCH 10/33] =?UTF-8?q?plan:=20set=20status=20reviewing=20?= =?UTF-8?q?=E2=80=94=20sub-phase=201.1=20gauntlet=20checkpoint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 2cb3353d530..fb8cb7f3771 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -100,7 +100,7 @@ v2/v3 paths. phase: "1: D — port v4 emitter (CODE)" sub_phase: "1.1 measurement_id contract" task: null -status: implementing +status: reviewing last_gate: null phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` From 878a17dee63e2cf50d4d65f3dcc92396b5a26c5a Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:41:30 -0400 Subject: [PATCH 11/33] plan: sub-phase 1.1 ledger entry + deferred carry-forward Record sub-phase 1.1 (measurement_id contract) as gauntlet pr-2 accepted (1 cycle), and park the 3 should-fix + 5 nit findings in Deferred work (golden-vector coverage gaps are out-of-monorepo-scope; CI wiring + xxhash importorskip + doc nits fold into sub-phase 1.3). Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index fb8cb7f3771..351348f2725 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -204,7 +204,24 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles #### Deferred work -- (none yet) +- **Sub-phase 1.1**, `scripts/test_measurement_id.py:41`, **should-fix**: module-level + `_load_port()` makes a missing `xxhash` a collection error rather than a skip. Deferral + rationale: address in sub-phase 1.3 (CI + test wiring) — add `pytest.importorskip("xxhash")` + and/or ensure CI supplies `xxhash` via `uv run --with`. +- **Sub-phase 1.1**, CI (no workflow runs the test), **should-fix**: the golden-parity test is + not wired into CI. Deferral rationale: that IS sub-phase 1.3's explicit scope. +- **Sub-phase 1.1**, `scripts/measurement_id_golden.json`, **should-fix**: golden vectors lack + NaN/Inf/-0.0 threshold and per-table empty-string coverage. Deferral rationale: OUT OF MONOREPO + SCOPE — the golden JSON is generated by the Rust test in the `vortex-data/benchmarks-website` + repo; coverage is added there and regenerated, never hand-edited in the monorepo. +- **Sub-phase 1.1**, `scripts/_measurement_id.py:15` + `scripts/test_measurement_id.py:34`, + **nit**: docstring misattributes the `measurement_id_golden_vectors` test to `db.rs` (it lives + in `tests/measurement_id_golden.rs`), plus a docstring phrasing fragment. Deferral rationale: + cosmetic doc nits carried verbatim from the source branch; fold into sub-phase 1.3 if it + touches these files. +- **Sub-phase 1.1**, `scripts/test_measurement_id.py` (`ensure_ascii`, 100-col), **nit**: add a + comment explaining `ensure_ascii=False` is load-bearing for the multibyte guard; 3 lines exceed + the 100-col style target (pass ruff's 120). Deferral rationale: cosmetic; optional cleanup in 1.3. --- @@ -212,4 +229,12 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles -(none yet — execution has not started) +### Phase 1: D — port v4 emitter (CODE) + +#### Sub-phase 1.1: measurement_id contract + +- **Shipped:** measurement_id Python port (`xxhash` XXH64) + 63-vector golden file + + golden-parity pytest, extracted verbatim from `f9b36ae3f`, docstrings repointed at the + extracted `vortex-data/benchmarks-website` repo; ruff-formatted to the repo's 120-col. +- **Gauntlet:** pr-2 / accepted (cycles: 1) +- **Deferred:** 5 items (see Carry-forward > Deferred work) From ad6f76f4ceeb4ebc3089821d45516db7866637e4 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:43:00 -0400 Subject: [PATCH 12/33] plan: advance to sub-phase 1.2 (Postgres writer) Sub-phase 1.1 complete + gauntlet-accepted; advance position to 1.2, status implementing. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 351348f2725..cff92ecb7d6 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -98,9 +98,9 @@ v2/v3 paths. ```yaml phase: "1: D — port v4 emitter (CODE)" -sub_phase: "1.1 measurement_id contract" +sub_phase: "1.2 Postgres writer" task: null -status: reviewing +status: implementing last_gate: null phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` From 74667a6ef762adc136290f41cf2756f213c03133 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:49:01 -0400 Subject: [PATCH 13/33] =?UTF-8?q?plan:=20amend=20sub-phase=201.2=20?= =?UTF-8?q?=E2=80=94=20self-contained=20schema=20fixture=20(migrations=20s?= =?UTF-8?q?tay=20out)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Class B plan-assumption violation resolved: the testcontainer writer test depends on the real migrations/ + migrate-schema.py + the website repo's schema.rs, all out of monorepo scope. Per user choice, adapt the test to a self-contained scripts/_v4_schema_fixture.sql (6 tables) applied directly, drop the migrate-runner dependency, and self-check SCHEMA_VERSION == 1 instead of reading the absent schema.rs. Migrations + migrate-schema.py remain out. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters-design.md | 11 +++++++++++ .big-plans/ct__bench-v4-emitters.md | 7 +++++++ 2 files changed, 18 insertions(+) diff --git a/.big-plans/ct__bench-v4-emitters-design.md b/.big-plans/ct__bench-v4-emitters-design.md index 5cef06693a1..029f8b1a51b 100644 --- a/.big-plans/ct__bench-v4-emitters-design.md +++ b/.big-plans/ct__bench-v4-emitters-design.md @@ -107,6 +107,17 @@ testcontainer deps via `--with` or the workspace.) - `scripts/measurement_id_golden.json` (new) — golden vectors (required, the test fails without it). - `scripts/test_measurement_id.py` (new) — golden-vector pytest (repoint docstring). - `scripts/test_post_ingest_postgres.py` (new) — testcontainers Postgres writer/upsert tests. + **ADAPTED (Class B amendment):** the branch version bootstraps its throwaway PG by applying the + real `migrations/` via `migrate-schema.py` and reads `benchmarks-website/server/src/schema.rs` + for a SCHEMA_VERSION lockstep check — all OUT of monorepo scope. Resolution (user-chosen): + give the test a SELF-CONTAINED schema fixture (a hand-written `.sql` creating the 6 tables the + emitter writes to, derived from the website repo's `migrations/001_initial_schema.sql`); rework + the `schema_conn` fixture to apply it (drop the `migrate-schema.py` runner dependency); and + adapt the SCHEMA_VERSION lockstep sub-test to self-check `post-ingest.py`'s `SCHEMA_VERSION == 1` + instead of reading the absent `schema.rs`. Migrations + `migrate-schema.py` stay OUT. +- `scripts/_v4_schema_fixture.sql` (new) — the self-contained 6-table schema fixture for the + testcontainer test (NOT the real migrations; a test-only DDL fixture). Drift from the real + migrations is managed like the SCHEMA_VERSION / column-list cross-repo contract. - `scripts/cross_check_python_writer.py` (new) — extra cross-check utility (later-phase extra). - `scripts/test_post_ingest_revalidate.py` (new) — revalidate-ping tests (later-phase extra). - `.github/workflows/bench.yml` — insert best-effort v4 step block after the v3 ingest step. diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index cff92ecb7d6..e1f8d462797 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -37,6 +37,13 @@ v2/v3 paths. wiring) PLUS the testcontainer writer tests (`test_post_ingest_postgres.py`) PLUS the extras (`cross_check_python_writer.py`, `test_post_ingest_revalidate.py`). Migrations + migrate-schema are OUT (extracted website repo owns schema/roles). See design spec § 3. +- **Decision (Class B amendment, sub-phase 1.2): testcontainer test uses a self-contained schema + fixture, migrations stay OUT.** The branch `test_post_ingest_postgres.py` depends on the real + `migrations/` + `migrate-schema.py` + `benchmarks-website/server/src/schema.rs` (all out of + monorepo scope), conflicting with "migrations OUT" + "testcontainer test IN". Resolved + (user-chosen): a hand-written `scripts/_v4_schema_fixture.sql` creates the 6 tables; the + `schema_conn` fixture applies it directly (no migrate-runner); the SCHEMA_VERSION lockstep + sub-test self-checks `post-ingest.py`'s `SCHEMA_VERSION == 1`. See design spec § 3. - **Decision: the v4 step is dormant until the switch is flipped.** Every v4 workflow step is gated on `vars.GH_BENCH_INGEST_ROLE_ARN != ''` with `continue-on-error: true`; merging D with the var unset ships dead-but-safe code. Setting the var (phase B) is the live cutover. From 1d73170e7e7874b4ad7e76893c4c0852fb1b0f2f Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:52:16 -0400 Subject: [PATCH 14/33] plan: task-plan for sub-phase 1.2 (Postgres writer) JIT writing-plans output for sub-phase 1.2: port the --postgres writer + revalidate test + cross-check utility, and adapt the testcontainer test to a self-contained schema fixture. 3 tasks. Signed-off-by: Connor Tsui --- ...h-v4-emitters--1-2-postgres-writer.plan.md | 221 ++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 .big-plans/ct__bench-v4-emitters--1-2-postgres-writer.plan.md diff --git a/.big-plans/ct__bench-v4-emitters--1-2-postgres-writer.plan.md b/.big-plans/ct__bench-v4-emitters--1-2-postgres-writer.plan.md new file mode 100644 index 00000000000..1ca502be8e5 --- /dev/null +++ b/.big-plans/ct__bench-v4-emitters--1-2-postgres-writer.plan.md @@ -0,0 +1,221 @@ +# Sub-phase 1.2 — Postgres writer Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Port the v4 `--postgres` writer into `scripts/post-ingest.py` (keeping the v3 `--server` path stdlib-only and intact), plus its tests + cross-check utility, adapting the testcontainer test to a self-contained schema fixture. + +**Architecture:** Extract-from-branch (`f9b36ae3f`) port. `post-ingest.py` applies cleanly (mainline never touched it). The testcontainer test is ADAPTED, not verbatim: it must not depend on the out-of-scope `migrations/` + `migrate-schema.py` + `benchmarks-website/server/src/schema.rs`; instead it applies a self-contained `scripts/_v4_schema_fixture.sql` and self-checks `SCHEMA_VERSION == 1`. + +**Tech Stack:** Python 3.11+, `psycopg[binary]`, `boto3` (RDS IAM token), `xxhash`, `testcontainers`, pytest, `uv run --no-project --with ...`, ruff, Docker (local + CI for the testcontainer suite). + +## Global Constraints + +- Extract files from `f9b36ae3f` via `git show f9b36ae3f: > `. Do NOT re-author the writer logic. +- The v3 `--server` path MUST stay importable + runnable under bare `python3` (stdlib only). The v4 deps (`psycopg`, `boto3`, `xxhash`) are imported LAZILY inside the `--postgres` code path only; `post-ingest.py`'s PEP-723 block stays `dependencies = []`. +- Repoint cross-repo doc references: `benchmarks-website/server/src/` and `benchmarks-website/AGENTS.md` no longer exist in this monorepo. Repoint to "the `vortex-data/benchmarks-website` repo's ``". Do NOT change any code/logic/values. +- `migrations/` + `migrate-schema.py` + `schema.rs` stay OUT of the monorepo (the testcontainer test is adapted to a self-contained fixture instead — Task 3). +- SPDX headers on every new file (`# SPDX-License-Identifier: Apache-2.0` / `# SPDX-FileCopyrightText: Copyright the Vortex contributors`; SQL uses `--` comment SPDX lines). +- Python lint: ruff `F,E,W,UP,I` clean, line-length 120. A whitespace-only `ruff format` of extracted files to the repo's 120-col is expected and acceptable (no logic/value change). Comments use `--`, never em dashes. +- This sub-phase touches ONLY: `scripts/post-ingest.py`, `scripts/test_post_ingest_postgres.py`, `scripts/test_post_ingest_revalidate.py`, `scripts/cross_check_python_writer.py`, `scripts/_v4_schema_fixture.sql`. Do NOT touch `ci.yml`, any workflow (sub-phase 1.3), or the 1.1 files. +- Commits: `git commit -F` with a heredoc (NEVER backticks or a `---` line in the message); sign off `Signed-off-by: Connor Tsui `. + +--- + +### Task 1: Port the `--postgres` writer into post-ingest.py + +**Files:** +- Modify (full replace): `scripts/post-ingest.py` (340 lines on develop -> 1252 lines from `f9b36ae3f`) + +**Interfaces:** +- Consumes: `scripts/_measurement_id.py` (ported in sub-phase 1.1) via the lazy `_measurement_id_module()` loader (`Path(__file__).parent / "_measurement_id.py"`). +- Produces (relied on by Tasks 2-3): `post_ingest.ingest_postgres(conn, commit, records) -> (inserted, updated)`, `post_ingest.connect_postgres(...)`, `post_ingest.refresh_site_cache(...)`, `post_ingest.SCHEMA_VERSION == 1`, and the `--server` / `--postgres` CLI dispatch. + +- [ ] **Step 1: Replace post-ingest.py with the branch version** + +```bash +cd "$(git rev-parse --show-toplevel)" +git show f9b36ae3f:scripts/post-ingest.py > scripts/post-ingest.py +``` + +- [ ] **Step 2: Verify the v3 `--server` path stays stdlib-only (the load-bearing safety property)** + +```bash +python3 scripts/post-ingest.py --help +``` + +Expected: prints usage showing BOTH `--server` and `--postgres` options, exits 0, with NO ImportError. This proves module import + the help/v3 path need no third-party packages (psycopg/boto3/xxhash are lazily imported only inside the `--postgres` branch). If this errors with a missing third-party module, the lazy-import boundary is broken -- STOP and report (do not "fix" by adding deps to PEP-723). + +- [ ] **Step 3: Repoint the cross-repo doc references** + +Grep first: `grep -n "benchmarks-website" scripts/post-ingest.py` (expect ~9 hits at lines near 41/43/47/65-67/328/527/909). For EACH hit, repoint the path token: `benchmarks-website/server/src/` -> ``the `vortex-data/benchmarks-website` repo's `server/src/` ``, and `benchmarks-website/AGENTS.md` -> ``the `vortex-data/benchmarks-website` repo's `AGENTS.md` ``. These are doc comments only -- do NOT touch any code, the `SCHEMA_VERSION = 1` constant, or the wire logic. Re-run `python3 scripts/post-ingest.py --help` after editing to confirm nothing broke. + +- [ ] **Step 4: Lint + format** + +```bash +uvx ruff check scripts/post-ingest.py +uvx ruff format --check scripts/post-ingest.py || uvx ruff format scripts/post-ingest.py +uvx ruff check scripts/post-ingest.py && uvx ruff format --check scripts/post-ingest.py +python3 -m py_compile scripts/post-ingest.py +``` + +Expected: ruff check clean; after the conditional `ruff format`, both `ruff check` and `ruff format --check` exit 0; py_compile exits 0. + +- [ ] **Step 5: Commit** + +```bash +git add scripts/post-ingest.py +git commit -F - <<'EOF' +scripts: add v4 --postgres IAM-auth upsert writer to post-ingest.py + +Port the v4 Postgres dual-write path (RDS IAM auth, verify-full TLS, NaN/Inf +guard, 5-table + commit-dim upsert in one transaction, best-effort revalidate) +from the v4 emitter branch, keeping the v3 --server path stdlib-only and intact. +v4 deps (psycopg, boto3, xxhash) are imported lazily inside the --postgres path. +Repoint docstrings to the extracted vortex-data/benchmarks-website repo. + +Signed-off-by: Connor Tsui +EOF +``` + +--- + +### Task 2: Port the revalidate test + cross-check utility + +**Files:** +- Create/Test: `scripts/test_post_ingest_revalidate.py` (262 lines; pure-stdlib, no Docker/psycopg) +- Create: `scripts/cross_check_python_writer.py` (247 lines; CLI cross-check utility) + +**Interfaces:** +- Consumes: `scripts/post-ingest.py` (Task 1) loaded by sibling path (`SCRIPTS_DIR / "post-ingest.py"`). + +- [ ] **Step 1: Extract both files from the branch** + +```bash +git show f9b36ae3f:scripts/test_post_ingest_revalidate.py > scripts/test_post_ingest_revalidate.py +git show f9b36ae3f:scripts/cross_check_python_writer.py > scripts/cross_check_python_writer.py +``` + +- [ ] **Step 2: Run the revalidate test (pure stdlib + pytest)** + +```bash +uv run --no-project --with pytest scripts/test_post_ingest_revalidate.py 2>/dev/null || uv run --no-project --with pytest --with xxhash pytest scripts/test_post_ingest_revalidate.py -q +``` + +Use this robust form: `uv run --no-project --with pytest pytest scripts/test_post_ingest_revalidate.py -q`. Expected: PASS. The test monkeypatches `urllib.request.urlopen` and exercises `refresh_site_cache` -- it loads `post-ingest.py` by path, whose module-level imports are stdlib, so no third-party deps are needed. If the import chain unexpectedly pulls a third-party module, add `--with xxhash` and report it as a concern (it would mean a module-level dep leaked). + +- [ ] **Step 3: Sanity-check the cross-check utility compiles and lints** + +```bash +python3 -m py_compile scripts/cross_check_python_writer.py +uvx ruff check scripts/cross_check_python_writer.py scripts/test_post_ingest_revalidate.py +uvx ruff format --check scripts/cross_check_python_writer.py scripts/test_post_ingest_revalidate.py || uvx ruff format scripts/cross_check_python_writer.py scripts/test_post_ingest_revalidate.py +uvx ruff check scripts/cross_check_python_writer.py scripts/test_post_ingest_revalidate.py && uvx ruff format --check scripts/cross_check_python_writer.py scripts/test_post_ingest_revalidate.py +``` + +Expected: py_compile exits 0; after the conditional format, ruff check + format --check both exit 0. (cross_check_python_writer.py is a CLI utility, not a test; py_compile + lint is the appropriate gate. Re-run the revalidate test if `ruff format` reformatted it.) + +- [ ] **Step 4: Commit** + +```bash +git add scripts/test_post_ingest_revalidate.py scripts/cross_check_python_writer.py +git commit -F - <<'EOF' +scripts: add revalidate test + python-writer cross-check utility + +Port the pure-stdlib revalidate-hook test (asserts refresh_site_cache sends the +bearer header and swallows every failure so it can never change the ingest exit +code) and the cross_check_python_writer.py utility (confirms the Python writer +recomputes the same measurement_id as seeded Rust-loaded rows and UPDATEs rather +than duplicating) from the v4 emitter branch. + +Signed-off-by: Connor Tsui +EOF +``` + +--- + +### Task 3: Self-contained schema fixture + adapted testcontainer test + +**Files:** +- Create: `scripts/_v4_schema_fixture.sql` (the 6-table DDL, derived from the website repo's `migrations/001_initial_schema.sql`) +- Create/Test: `scripts/test_post_ingest_postgres.py` (1309 lines from the branch, ADAPTED) + +**Interfaces:** +- Consumes: `post_ingest.ingest_postgres` / `connect_postgres` (Task 1); the schema fixture (this task). + +- [ ] **Step 1: Create the self-contained schema fixture** + +Read the reference DDL (READ-ONLY, does not land here): `/Users/connor/spiral/vortex-data/benchmarks-website/migrations/001_initial_schema.sql`. Create `scripts/_v4_schema_fixture.sql` containing the two SPDX `--` header lines, a comment noting it is a TEST-ONLY fixture mirroring that repo's `migrations/001_initial_schema.sql` (drift managed like the SCHEMA_VERSION / column-list cross-repo contract), then the six `CREATE TABLE IF NOT EXISTS` statements (`commits`, `query_measurements`, `compression_times`, `compression_sizes`, `random_access_times`, `vector_search_runs`) and their `CREATE INDEX IF NOT EXISTS` statements, copied verbatim (columns/types/PKs/nullability EXACTLY as in 001 -- `DOUBLE PRECISION`, `BIGINT[]`, `TIMESTAMPTZ`, etc.). Do NOT include the role/grant migrations (002-007); the test connects as the container superuser, so no roles are needed. + +- [ ] **Step 2: Extract the testcontainer test from the branch** + +```bash +git show f9b36ae3f:scripts/test_post_ingest_postgres.py > scripts/test_post_ingest_postgres.py +``` + +- [ ] **Step 3: Run it unmodified to confirm it fails on the out-of-scope deps (red)** + +```bash +uv run --no-project --with pytest --with 'psycopg[binary]' --with boto3 --with xxhash --with testcontainers pytest scripts/test_post_ingest_postgres.py -q 2>&1 | tail -25 +``` + +Expected: a collection/setup ERROR -- the module tries `_load_module("migrate-schema.py", ...)` (line ~77) and/or reads `migrations/` (line ~45), which do not exist in the monorepo. This confirms the adaptation in Step 4 is required. + +- [ ] **Step 4: Adapt the test off the out-of-scope dependencies** + +Grep the ported file for every out-of-scope reference: `grep -n "migrate_runner\|migrate-schema\|REPO_MIGRATIONS_DIR\|migrations\|schema.rs\|benchmarks-website" scripts/test_post_ingest_postgres.py`. Apply these edits (and any others the grep surfaces), changing ONLY schema-bootstrap plumbing, never the assertions/test bodies: + +1. Remove the migrate-runner module load (line ~77: `migrate_runner = _load_module("migrate-schema.py", "migrate_schema")`) and the `REPO_MIGRATIONS_DIR = REPO_ROOT / "migrations"` binding (line ~45). Keep `REPO_ROOT` only if still used elsewhere; if it becomes unused, remove it (ruff F841/unused will flag). +2. Add a fixture-path constant near the other path constants, e.g. `_SCHEMA_FIXTURE = SCRIPTS_DIR / "_v4_schema_fixture.sql"`. +3. In the `schema_conn` fixture, replace `migrate_runner.apply(conn, REPO_MIGRATIONS_DIR)` (line ~285) with applying the fixture directly, e.g.: + ```python + with conn.cursor() as cur: + cur.execute(_SCHEMA_FIXTURE.read_text()) + ``` + (The fixture's `CREATE TABLE/INDEX IF NOT EXISTS` are idempotent; the existing scrub-then-apply flow is preserved.) +4. Adapt the SCHEMA_VERSION lockstep test (lines ~849-857) that reads `benchmarks-website/server/src/schema.rs`: replace the file read + regex with a self-check that `post_ingest.SCHEMA_VERSION == 1` (the monorepo cannot read the website repo's `schema.rs`; the cross-repo lockstep is enforced by CONTRACT.md, not testable here). Keep the test function (rename its docstring to reflect the self-check) so the constant is still guarded. +5. Update the module docstring (lines ~5-9) to say the schema comes from the self-contained `_v4_schema_fixture.sql`, not the real `migrations/` via a migrate runner. + +- [ ] **Step 5: Run the adapted testcontainer suite under Docker (green -- the load-bearing pre-merge gate)** + +```bash +docker info >/dev/null 2>&1 && echo "docker OK" || echo "DOCKER MISSING -- escalate" +uv run --no-project --with pytest --with 'psycopg[binary]' --with boto3 --with xxhash --with testcontainers pytest scripts/test_post_ingest_postgres.py -q 2>&1 | tail -30 +``` + +Expected: all tests PASS (the integration tests spin a `postgres:16-alpine` container, apply the fixture, and exercise the real upsert path; the mocked unit tests cover `connect_postgres`'s IAM/TLS/role enforcement). This is the load-bearing pre-merge confidence gate -- it proves the writer's upsert/transaction/measurement_id/NaN-guard behavior against a real Postgres with ZERO prod or develop dependency. If Docker is missing, escalate (do not let the suite silently skip -- this gate must actually run before merge). + +- [ ] **Step 6: Lint + format the test and fixture** + +```bash +uvx ruff check scripts/test_post_ingest_postgres.py +uvx ruff format --check scripts/test_post_ingest_postgres.py || uvx ruff format scripts/test_post_ingest_postgres.py +uvx ruff check scripts/test_post_ingest_postgres.py && uvx ruff format --check scripts/test_post_ingest_postgres.py +``` + +Expected: ruff check + format --check both exit 0. (The `.sql` fixture is not ruff-governed.) If `ruff format` changed the test, re-run Step 5 to confirm it still passes. + +- [ ] **Step 7: Commit** + +```bash +git add scripts/_v4_schema_fixture.sql scripts/test_post_ingest_postgres.py +git commit -F - <<'EOF' +scripts: add adapted testcontainer writer test + self-contained schema fixture + +Port the testcontainer Postgres writer test and adapt it off the out-of-scope +migrations/ + migrate-schema.py + benchmarks-website/server/src/schema.rs: apply +a self-contained scripts/_v4_schema_fixture.sql (the 6-table DDL mirroring the +website repo's migrations/001) and self-check SCHEMA_VERSION == 1. The suite +spins postgres:16-alpine and exercises the real upsert/transaction path, the +load-bearing pre-merge confidence gate with no prod or develop dependency. + +Signed-off-by: Connor Tsui +EOF +``` + +--- + +## Self-Review + +- **Spec coverage:** post-ingest.py `--postgres` writer ported with v3 path intact (Task 1); revalidate test + cross-check utility ported (Task 2); testcontainer test adapted to the self-contained fixture + SCHEMA_VERSION self-check (Task 3). Out-of-scope (migrations/, migrate-schema.py, ci.yml, workflows) untouched. +- **Placeholder scan:** none -- every step has exact commands or exact edit instructions with line anchors. +- **Type consistency:** `ingest_postgres` / `connect_postgres` / `SCHEMA_VERSION` / `refresh_site_cache` names match across Task 1 (producer) and Tasks 2-3 (consumers); the fixture's table/column names match the upsert column lists in post-ingest.py and the golden `table` values from sub-phase 1.1. From 5a4d8aec059541a40b5972b1b5786e8669d84056 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:54:58 -0400 Subject: [PATCH 15/33] scripts: add v4 --postgres IAM-auth upsert writer to post-ingest.py Port the v4 Postgres dual-write path (RDS IAM auth, verify-full TLS, NaN/Inf guard, 5-table + commit-dim upsert in one transaction, best-effort revalidate) from the v4 emitter branch, keeping the v3 --server path stdlib-only and intact. v4 deps (psycopg, boto3, xxhash) are imported lazily inside the --postgres path. Repoint docstrings to the extracted vortex-data/benchmarks-website repo. Signed-off-by: Connor Tsui --- scripts/post-ingest.py | 967 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 940 insertions(+), 27 deletions(-) diff --git a/scripts/post-ingest.py b/scripts/post-ingest.py index 2a72f975d95..38c352dea3d 100755 --- a/scripts/post-ingest.py +++ b/scripts/post-ingest.py @@ -7,29 +7,44 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright the Vortex contributors -"""Wrap a `--gh-json-v3` JSONL file in an envelope and POST to /api/ingest. +"""Ingest a `--gh-json-v3` JSONL file into the benchmarks store. -Reads bare v3 records from a JSONL file produced by `vortex-bench --gh-json-v3`, -fills the `commit` envelope by shelling out to `git show`, and POSTs the -envelope to `/api/ingest` with a bearer token. +Reads bare v3 records from a JSONL file produced by `vortex-bench --gh-json-v3` +and fills the `commit` fields by shelling out to `git show`. Two mutually +exclusive ingest modes select the destination substrate: -Standard library only -- urllib, json, subprocess. The default envelope size -(60 MiB, just under the server's 64 MiB body limit) is sized so a single -JSONL run normally posts in one envelope -- preserving the "per-file -all-or-nothing" contract the server documents. If the JSONL is large enough -that splitting kicks in, the script emits a warning and proceeds with the +- `--server ` (v3): wraps the records in an envelope and POSTs to + `/api/ingest` with a bearer token. Standard library only -- urllib, + json, subprocess. +- `--postgres ` (v4): computes the server-internal `measurement_id` + locally (via `_measurement_id.py`) and upserts directly into the RDS + Postgres tables with `INSERT ... ON CONFLICT (measurement_id) DO UPDATE`, + over a verify-full TLS connection, authenticating with an RDS IAM auth token + when the DSN carries no password. Requires `psycopg`, `boto3`, and `xxhash` + from the project environment; these are imported lazily inside the + `--postgres` code path so the v3 `--server` path stays standard-library-only + under a bare `python3` (CI invokes `python3 scripts/post-ingest.py`, not + `uv run`, and the v3 path is in production until the Phase 5 cutover). The + PEP 723 metadata block below intentionally keeps `dependencies = []` for that + reason; run the `--postgres` mode from the repo's uv environment. + +The default envelope size (60 MiB, just under the server's 64 MiB body limit) +is sized so a single JSONL run normally posts in one envelope -- preserving the +"per-file all-or-nothing" contract the server documents. If the JSONL is large +enough that splitting kicks in, the script emits a warning and proceeds with the chunked semantics (per-chunk commit, mid-chunk failure leaves earlier chunks -ingested; subsequent retries re-upsert via the server's ON CONFLICT -idempotency on `measurement_id`). +ingested; subsequent retries re-upsert via the server's ON CONFLICT idempotency +on `measurement_id`). The `--postgres` mode applies a whole JSONL file in one +transaction (all-or-nothing), with no chunking. Wire-contract pointers (kept in sync as a coordinated change per -`benchmarks-website/AGENTS.md`): +the `vortex-data/benchmarks-website` repo's `AGENTS.md`): -- `benchmarks-website/server/src/records.rs` - envelope + per-record wire +- the `vortex-data/benchmarks-website` repo's `server/src/records.rs` - envelope + per-record wire shapes that the server deserializes. - `vortex-bench/src/v3.rs` - bare-record producer that writes the JSONL this script wraps. -- `benchmarks-website/server/src/schema.rs` - `SCHEMA_VERSION` source of +- the `vortex-data/benchmarks-website` repo's `server/src/schema.rs` - `SCHEMA_VERSION` source of truth that the `SCHEMA_VERSION` constant below MUST equal at every bump. """ @@ -37,17 +52,19 @@ import argparse import json +import math import os import subprocess import sys import urllib.error import urllib.request +from concurrent.futures import ThreadPoolExecutor from datetime import UTC, datetime from pathlib import Path -# MUST equal `benchmarks-website/server/src/schema.rs::SCHEMA_VERSION`. +# MUST equal the `vortex-data/benchmarks-website` repo's `server/src/schema.rs::SCHEMA_VERSION`. # Bumping this is a coordinated change across schema.rs, records.rs, v3.rs, -# and this script. See `benchmarks-website/AGENTS.md` ("Wire shapes are a +# and this script. See the `vortex-data/benchmarks-website` repo's `AGENTS.md` ("Wire shapes are a # coordinated change") for the full list of coupled sites. SCHEMA_VERSION = 1 # Default sized to fit comfortably under the server's 64 MiB ingest body @@ -59,17 +76,45 @@ def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( - description="POST a v3 JSONL records file to /api/ingest.", + description=( + "Ingest a v3 JSONL records file: --server POSTs a v3 envelope to " + "/api/ingest; --postgres upserts directly into the RDS Postgres tables." + ), ) parser.add_argument( "jsonl_path", type=Path, help="Path to the JSONL file written by vortex-bench --gh-json-v3.", ) - parser.add_argument( + mode = parser.add_mutually_exclusive_group(required=True) + mode.add_argument( "--server", - required=True, - help="Server base URL, e.g. http://localhost:8080.", + help=( + "v3 mode: server base URL, e.g. http://localhost:8080. Wraps the " + "records in an envelope and POSTs to /api/ingest with a " + "bearer token. Mutually exclusive with --postgres." + ), + ) + mode.add_argument( + "--postgres", + metavar="DSN", + help=( + "v4 mode: libpq DSN for the RDS Postgres ingest target, e.g. " + "'postgresql://bench_ingest@host:5432/benchmarks?sslmode=verify-full" + "&sslrootcert=/path/rds-ca.pem'. Computes measurement_id locally and " + "upserts via INSERT ... ON CONFLICT DO UPDATE. If the DSN carries no " + "password, an RDS IAM auth token is minted for the DSN's user. " + "Mutually exclusive with --server." + ), + ) + parser.add_argument( + "--region", + default=None, + help=( + "AWS region for RDS IAM token minting (--postgres mode). Precedence: " + "this explicit --region, then the boto3 session region, then the region " + "parsed from the RDS hostname." + ), ) parser.add_argument( "--commit-sha", @@ -78,8 +123,10 @@ def parse_args() -> argparse.Namespace: ) parser.add_argument( "--benchmark-id", - required=True, - help="Run identifier echoed back in run_meta.benchmark_id.", + default=None, + help="Run identifier echoed back in run_meta.benchmark_id. Required for " + "--server mode; unused by --postgres mode (the v4 tables have no " + "benchmark_id column).", ) parser.add_argument( "--token-env", @@ -113,9 +160,11 @@ def parse_args() -> argparse.Namespace: def read_records(path: Path) -> list[dict]: + # The JSONL comes from the project's own `vortex-bench` CI on the same run (trusted UTF-8 + # input); a malformed line still fails loud with `path:line` context for debuggability. records: list[dict] = [] - with path.open("r", encoding="utf-8") as fp: - for line_no, line in enumerate(fp, start=1): + with path.open(encoding="utf-8") as f: + for line_no, line in enumerate(f, start=1): line = line.strip() if not line: continue @@ -127,7 +176,10 @@ def read_records(path: Path) -> list[dict]: def git_show_field(sha: str, fmt: str, cwd: Path | None) -> str: - """Run `git show -s --format= ` and return its stdout (stripped).""" + """Run `git show -s --format= ` and return its stdout (stripped). + + The metadata comes from the project's own (UTF-8) git history. + """ result = subprocess.run( ["git", "show", "-s", f"--format={fmt}", sha], cwd=cwd, @@ -265,9 +317,863 @@ def post(server: str, body: bytes, token: str, timeout: float) -> tuple[int, byt return exc.code, exc.read() -def main() -> int: - args = parse_args() +# -------------------------------------------------------------------------- +# Postgres dual-write mode (--postgres) +# -------------------------------------------------------------------------- +# The v4 ingest path. `psycopg`, `boto3`, and `_measurement_id` (which pulls in +# `xxhash`) are imported lazily inside the functions below so the v3 `--server` +# path above stays standard-library-only under a bare `python3`. + +# Per-`kind` record field sets, mirroring the `#[serde(deny_unknown_fields)]` +# structs in the `vortex-data/benchmarks-website` repo's `server/src/records.rs`. `required` plus +# `optional` is the exact set of fields a record of that `kind` may carry +# (besides `kind` itself); an unknown field is rejected loudly, preserving the +# v3 server's deny_unknown_fields behavior so producer/schema drift surfaces +# instead of silently dropping data. `optional` fields default to NULL. +_RECORD_FIELDS: dict[str, tuple[frozenset[str], frozenset[str]]] = { + "query_measurement": ( + frozenset( + { + "commit_sha", + "dataset", + "query_idx", + "storage", + "engine", + "format", + "value_ns", + "all_runtimes_ns", + } + ), + frozenset( + { + "dataset_variant", + "scale_factor", + "peak_physical", + "peak_virtual", + "physical_delta", + "virtual_delta", + "env_triple", + } + ), + ), + "compression_time": ( + frozenset({"commit_sha", "dataset", "format", "op", "value_ns", "all_runtimes_ns"}), + frozenset({"dataset_variant", "env_triple"}), + ), + "compression_size": ( + frozenset({"commit_sha", "dataset", "format", "value_bytes"}), + frozenset({"dataset_variant"}), + ), + "random_access_time": ( + frozenset({"commit_sha", "dataset", "format", "value_ns", "all_runtimes_ns"}), + frozenset({"env_triple"}), + ), + "vector_search_run": ( + frozenset( + { + "commit_sha", + "dataset", + "layout", + "flavor", + "threshold", + "value_ns", + "all_runtimes_ns", + "matches", + "rows_scanned", + "bytes_scanned", + "iterations", + } + ), + frozenset({"env_triple"}), + ), +} + +_MEASUREMENT_ID_MODULE = None + + +def _measurement_id_module(): + """Lazily load `scripts/_measurement_id.py` by path (cached). + + Loaded by file path rather than `import _measurement_id` so it resolves + regardless of cwd / `sys.path` (the test harness loads sibling scripts the + same way). Importing it pulls in `xxhash`, so it only happens here on the + `--postgres` path. + """ + global _MEASUREMENT_ID_MODULE + if _MEASUREMENT_ID_MODULE is None: + import importlib.util + + path = Path(__file__).resolve().parent / "_measurement_id.py" + spec = importlib.util.spec_from_file_location("_measurement_id", path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + _MEASUREMENT_ID_MODULE = module + return _MEASUREMENT_ID_MODULE + + +def _validate_record_fields(record: object, index: int) -> str: + """Validate a record's `kind` and field set; return the `kind`. + + Mirrors the server's deny_unknown_fields + required-field deserialization: + a non-object record, an unknown `kind`, an unknown field, or a missing + required field is a loud error keyed by the record's index. + """ + if not isinstance(record, dict): + raise SystemExit(f"record {index}: expected a JSON object, got {type(record).__name__}") + kind = record.get("kind") + # `isinstance(kind, str)` first: a non-scalar kind (e.g. a list) is unhashable + # and would raise TypeError at the `in` membership check rather than this + # controlled record-indexed error. + if not isinstance(kind, str) or kind not in _RECORD_FIELDS: + raise SystemExit(f"record {index}: unknown kind {kind!r}; expected one of {sorted(_RECORD_FIELDS)}") + required, optional = _RECORD_FIELDS[kind] + present = set(record) - {"kind"} + unknown = present - required - optional + if unknown: + raise SystemExit(f"record {index} ({kind}): unknown field(s) {sorted(unknown)} (deny_unknown_fields)") + missing = required - present + if missing: + raise SystemExit(f"record {index} ({kind}): missing required field(s) {sorted(missing)}") + return kind + + +def _require_finite(value: object, field: str, kind: str, index: int) -> None: + """Raise loudly unless `value` is a finite number. + + Guards every f64 dim that feeds a `measurement_id` hash (currently only + `vector_search_run.threshold`). A NaN/Inf would hash differently across the + Rust `to_bits()` and Python `struct.pack(' None: + """Raise loudly unless `value` is a plain (non-bool) integer within i32/i64 range. + + The integer columns bind straight to INTEGER/BIGINT; psycopg adapts a Python + float to float8 and Postgres assignment-casts (rounds) it, so a JSON float + would silently persist a rounded value where the v3 server's serde `i32`/`i64` + rejects it. An out-of-range integer would otherwise fail late as an uncaught + `struct.error` (i32 hash dims via `_write_i32`) or a raw Postgres 22003 + overflow; validate the type AND width here so a malformed scalar fails loud + (record-indexed) instead, matching the v3 serde boundary. + """ + lo, hi = (_INT32_MIN, _INT32_MAX) if bits == 32 else (_INT64_MIN, _INT64_MAX) + if isinstance(value, bool) or not isinstance(value, int): + raise SystemExit(f"record {index} ({kind}): {field} must be an integer, got {value!r}") + if not (lo <= value <= hi): + raise SystemExit(f"record {index} ({kind}): {field}={value!r} is out of int{bits} range") + + +def _require_int_list(value: object, field: str, kind: str, index: int) -> None: + """Raise loudly unless `value` is a JSON array of plain (non-bool) i64 integers. + + Guards the `all_runtimes_ns` -> `bigint[]` bind: the explicit `::bigint[]` + cast is permissive in ways the v3 server's `Vec` serde is not. psycopg + sends the string `"{}"` as text and the cast parses it into an empty array, a + `[1, null]` list adapts to `{1,NULL}`, and an out-of-i64 element hits a raw + Postgres 22003 -- each diverges from or fails differently than the v3 path. + Validate the element type AND i64 range so a malformed value fails loud. + """ + if not isinstance(value, list) or any(isinstance(x, bool) or not isinstance(x, int) for x in value): + raise SystemExit(f"record {index} ({kind}): {field} must be a JSON array of integers, got {value!r}") + if any(not (_INT64_MIN <= x <= _INT64_MAX) for x in value): + raise SystemExit(f"record {index} ({kind}): {field} has an element out of int64 range") + + +def _require_str(value: object, field: str, kind: str, index: int) -> None: + """Raise loudly unless `value` is a string. Mirrors the v3 serde `String` fields.""" + if not isinstance(value, str): + raise SystemExit(f"record {index} ({kind}): {field} must be a string, got {value!r}") + + +def _require_opt_str(value: object, field: str, kind: str, index: int) -> None: + """Raise loudly unless `value` is a string or null. Mirrors `Option`.""" + if value is not None and not isinstance(value, str): + raise SystemExit(f"record {index} ({kind}): {field} must be a string or null, got {value!r}") + + +def _memory_quartet_consistent(r: dict) -> bool: + """The four `query_measurements` memory columns are all set or all absent. + + Mirrors `ingest.rs::memory_quartet_consistent`: a partial quartet is a + validation error, not a half-populated row. + """ + present = [ + r.get("peak_physical") is not None, + r.get("peak_virtual") is not None, + r.get("physical_delta") is not None, + r.get("virtual_delta") is not None, + ] + return not any(present) or all(present) + + +# Per-kind field -> type token, mirroring the v3 server's typed serde boundary in +# `records.rs`. The direct-to-Postgres writer can no longer lean on server-side +# deserialization, so every field is type/range-validated here. Tokens: +# "str" required String "opt_str" Option +# "i32" required i32 "i64" required i64 +# "opt_i64" Option "i64_list" Vec (all_runtimes_ns) +# "f64" finite f64 (threshold). +# Required tokens (every non-opt_*) line up with `_RECORD_FIELDS` required sets, so +# `record[field]` is always present by the time `_validate_record_values` runs. +_FIELD_TYPES: dict[str, tuple[tuple[str, str], ...]] = { + "query_measurement": ( + ("commit_sha", "str"), + ("dataset", "str"), + ("dataset_variant", "opt_str"), + ("scale_factor", "opt_str"), + ("query_idx", "i32"), + ("storage", "str"), + ("engine", "str"), + ("format", "str"), + ("value_ns", "i64"), + ("all_runtimes_ns", "i64_list"), + ("peak_physical", "opt_i64"), + ("peak_virtual", "opt_i64"), + ("physical_delta", "opt_i64"), + ("virtual_delta", "opt_i64"), + ("env_triple", "opt_str"), + ), + "compression_time": ( + ("commit_sha", "str"), + ("dataset", "str"), + ("dataset_variant", "opt_str"), + ("format", "str"), + ("op", "str"), + ("value_ns", "i64"), + ("all_runtimes_ns", "i64_list"), + ("env_triple", "opt_str"), + ), + "compression_size": ( + ("commit_sha", "str"), + ("dataset", "str"), + ("dataset_variant", "opt_str"), + ("format", "str"), + ("value_bytes", "i64"), + ), + "random_access_time": ( + ("commit_sha", "str"), + ("dataset", "str"), + ("format", "str"), + ("value_ns", "i64"), + ("all_runtimes_ns", "i64_list"), + ("env_triple", "opt_str"), + ), + "vector_search_run": ( + ("commit_sha", "str"), + ("dataset", "str"), + ("layout", "str"), + ("flavor", "str"), + ("threshold", "f64"), + ("value_ns", "i64"), + ("all_runtimes_ns", "i64_list"), + ("matches", "i64"), + ("rows_scanned", "i64"), + ("bytes_scanned", "i64"), + ("iterations", "i32"), + ("env_triple", "opt_str"), + ), +} + + +def _validate_record_values(record: dict, kind: str, index: int) -> None: + """Validate every field's type/range against the v3 server's serde boundary. + + Runs in `ingest_postgres`'s loop (where the record index is known) so every + failure is reported as `record {index} ({kind}): ...`, matching the v3 + server's indexed per-record errors. Drives field type/range checks off + `_FIELD_TYPES`, then applies the semantic checks the type alone does not cover + (the storage enum + memory quartet for query_measurements). + """ + for field, typ in _FIELD_TYPES[kind]: + if typ == "str": + _require_str(record[field], field, kind, index) + elif typ == "opt_str": + _require_opt_str(record.get(field), field, kind, index) + elif typ == "i32": + _require_int(record[field], field, kind, index, bits=32) + elif typ == "i64": + _require_int(record[field], field, kind, index, bits=64) + elif typ == "opt_i64": + if record.get(field) is not None: + _require_int(record[field], field, kind, index, bits=64) + elif typ == "i64_list": + _require_int_list(record[field], field, kind, index) + elif typ == "f64": + _require_finite(record[field], field, kind, index) + + if kind == "query_measurement": + if record["storage"] not in ("nvme", "s3"): + raise SystemExit( + f"record {index} (query_measurement): storage must be 'nvme' or 's3', got {record['storage']!r}" + ) + if not _memory_quartet_consistent(record): + raise SystemExit( + f"record {index} (query_measurement): memory fields must be populated together (all four or none)" + ) + + +def _upsert_returning_was_update(conn, sql: str, params: tuple) -> bool: + """Run an `INSERT ... ON CONFLICT DO UPDATE ... RETURNING (xmax = 0)` upsert + and return whether the row was an update (vs a fresh insert). + + Classifies inserted-vs-updated atomically from the upsert itself: a fresh + INSERT leaves the new tuple's system column `xmax = 0`, while `ON CONFLICT + DO UPDATE` stamps `xmax` with the current transaction id. A preflight SELECT + (the prior approach) would race the upsert under concurrent re-ingest of the + same `measurement_id`, miscounting a concurrent loser as inserted; deriving + the flag from the single atomic statement removes that window. A duplicate + `measurement_id` within ONE transaction is also handled correctly: the second + upsert locks the tuple the first created (stamping its xmax), so it is + classified as an update, matching the v3 server. `sql` must end with + `RETURNING (xmax = 0) AS inserted`. + """ + row = conn.execute(sql, params).fetchone() + return not row[0] + + +def _insert_query_measurement(conn, mid_mod, r: dict) -> bool: + """Upsert a `query_measurements` row. Mirrors `ingest.rs::insert_query_measurement`. + + Record values are validated by `_validate_record_values` before dispatch. + """ + mid = mid_mod.measurement_id_query( + commit_sha=r["commit_sha"], + dataset=r["dataset"], + dataset_variant=r.get("dataset_variant"), + scale_factor=r.get("scale_factor"), + query_idx=r["query_idx"], + storage=r["storage"], + engine=r["engine"], + format=r["format"], + ) + return _upsert_returning_was_update( + conn, + """ + INSERT INTO query_measurements ( + measurement_id, commit_sha, dataset, dataset_variant, scale_factor, + query_idx, storage, engine, format, + value_ns, all_runtimes_ns, + peak_physical, peak_virtual, physical_delta, virtual_delta, + env_triple, commit_timestamp + ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s::bigint[], %s, %s, %s, %s, %s, + (SELECT timestamp FROM commits WHERE commit_sha = %s)) + ON CONFLICT (measurement_id) DO UPDATE SET + commit_sha = excluded.commit_sha, + value_ns = excluded.value_ns, + all_runtimes_ns = excluded.all_runtimes_ns, + peak_physical = excluded.peak_physical, + peak_virtual = excluded.peak_virtual, + physical_delta = excluded.physical_delta, + virtual_delta = excluded.virtual_delta, + env_triple = excluded.env_triple, + commit_timestamp = excluded.commit_timestamp + RETURNING (xmax = 0) AS inserted + """, + ( + mid, + r["commit_sha"], + r["dataset"], + r.get("dataset_variant"), + r.get("scale_factor"), + r["query_idx"], + r["storage"], + r["engine"], + r["format"], + r["value_ns"], + r["all_runtimes_ns"], + r.get("peak_physical"), + r.get("peak_virtual"), + r.get("physical_delta"), + r.get("virtual_delta"), + r.get("env_triple"), + # The denormalized `commit_timestamp` (migration 006) is resolved from the + # `commits` row this same transaction upserted first, so the read path's + # latest-per-series summary never sees a NULL from this writer. + r["commit_sha"], + ), + ) + + +def _insert_compression_time(conn, mid_mod, r: dict) -> bool: + """Upsert a `compression_times` row. Mirrors `ingest.rs::insert_compression_time`.""" + mid = mid_mod.measurement_id_compression_time( + commit_sha=r["commit_sha"], + dataset=r["dataset"], + dataset_variant=r.get("dataset_variant"), + format=r["format"], + op=r["op"], + ) + return _upsert_returning_was_update( + conn, + """ + INSERT INTO compression_times ( + measurement_id, commit_sha, dataset, dataset_variant, + format, op, value_ns, all_runtimes_ns, env_triple + ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s::bigint[], %s) + ON CONFLICT (measurement_id) DO UPDATE SET + commit_sha = excluded.commit_sha, + value_ns = excluded.value_ns, + all_runtimes_ns = excluded.all_runtimes_ns, + env_triple = excluded.env_triple + RETURNING (xmax = 0) AS inserted + """, + ( + mid, + r["commit_sha"], + r["dataset"], + r.get("dataset_variant"), + r["format"], + r["op"], + r["value_ns"], + r["all_runtimes_ns"], + r.get("env_triple"), + ), + ) + + +def _insert_compression_size(conn, mid_mod, r: dict) -> bool: + """Upsert a `compression_sizes` row. Mirrors `ingest.rs::insert_compression_size`.""" + mid = mid_mod.measurement_id_compression_size( + commit_sha=r["commit_sha"], + dataset=r["dataset"], + dataset_variant=r.get("dataset_variant"), + format=r["format"], + ) + return _upsert_returning_was_update( + conn, + """ + INSERT INTO compression_sizes ( + measurement_id, commit_sha, dataset, dataset_variant, + format, value_bytes + ) VALUES (%s, %s, %s, %s, %s, %s) + ON CONFLICT (measurement_id) DO UPDATE SET + commit_sha = excluded.commit_sha, + value_bytes = excluded.value_bytes + RETURNING (xmax = 0) AS inserted + """, + ( + mid, + r["commit_sha"], + r["dataset"], + r.get("dataset_variant"), + r["format"], + r["value_bytes"], + ), + ) + + +def _insert_random_access(conn, mid_mod, r: dict) -> bool: + """Upsert a `random_access_times` row. Mirrors `ingest.rs::insert_random_access`.""" + mid = mid_mod.measurement_id_random_access( + commit_sha=r["commit_sha"], + dataset=r["dataset"], + format=r["format"], + ) + return _upsert_returning_was_update( + conn, + """ + INSERT INTO random_access_times ( + measurement_id, commit_sha, dataset, format, + value_ns, all_runtimes_ns, env_triple + ) VALUES (%s, %s, %s, %s, %s, %s::bigint[], %s) + ON CONFLICT (measurement_id) DO UPDATE SET + commit_sha = excluded.commit_sha, + value_ns = excluded.value_ns, + all_runtimes_ns = excluded.all_runtimes_ns, + env_triple = excluded.env_triple + RETURNING (xmax = 0) AS inserted + """, + ( + mid, + r["commit_sha"], + r["dataset"], + r["format"], + r["value_ns"], + r["all_runtimes_ns"], + r.get("env_triple"), + ), + ) + +def _insert_vector_search(conn, mid_mod, r: dict) -> bool: + """Upsert a `vector_search_runs` row. Mirrors `ingest.rs::insert_vector_search`. + + `threshold` is validated finite by `_validate_record_values` before dispatch. + """ + threshold = float(r["threshold"]) + mid = mid_mod.measurement_id_vector_search( + commit_sha=r["commit_sha"], + dataset=r["dataset"], + layout=r["layout"], + flavor=r["flavor"], + threshold=threshold, + ) + return _upsert_returning_was_update( + conn, + """ + INSERT INTO vector_search_runs ( + measurement_id, commit_sha, dataset, layout, flavor, threshold, + value_ns, all_runtimes_ns, matches, rows_scanned, bytes_scanned, + iterations, env_triple + ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s::bigint[], %s, %s, %s, %s, %s) + ON CONFLICT (measurement_id) DO UPDATE SET + commit_sha = excluded.commit_sha, + value_ns = excluded.value_ns, + all_runtimes_ns = excluded.all_runtimes_ns, + matches = excluded.matches, + rows_scanned = excluded.rows_scanned, + bytes_scanned = excluded.bytes_scanned, + iterations = excluded.iterations, + env_triple = excluded.env_triple + RETURNING (xmax = 0) AS inserted + """, + ( + mid, + r["commit_sha"], + r["dataset"], + r["layout"], + r["flavor"], + threshold, + r["value_ns"], + r["all_runtimes_ns"], + r["matches"], + r["rows_scanned"], + r["bytes_scanned"], + r["iterations"], + r.get("env_triple"), + ), + ) + + +# Dispatch from a record's `kind` to its per-table upsert. Keyed identically to +# `_RECORD_FIELDS`; the two maps are wired together when adding a fact table. +_APPLY_RECORD = { + "query_measurement": _insert_query_measurement, + "compression_time": _insert_compression_time, + "compression_size": _insert_compression_size, + "random_access_time": _insert_random_access, + "vector_search_run": _insert_vector_search, +} + + +def _upsert_commit(conn, commit: dict) -> None: + """Upsert the `commits` dim row. Mirrors `ingest.rs::upsert_commit`.""" + conn.execute( + """ + INSERT INTO commits ( + commit_sha, timestamp, message, author_name, author_email, + committer_name, committer_email, tree_sha, url + ) VALUES (%s, %s::timestamptz, %s, %s, %s, %s, %s, %s, %s) + ON CONFLICT (commit_sha) DO UPDATE SET + timestamp = excluded.timestamp, + message = excluded.message, + author_name = excluded.author_name, + author_email = excluded.author_email, + committer_name = excluded.committer_name, + committer_email = excluded.committer_email, + tree_sha = excluded.tree_sha, + url = excluded.url + """, + ( + commit["sha"], + commit["timestamp"], + commit["message"], + commit["author_name"], + commit["author_email"], + commit["committer_name"], + commit["committer_email"], + commit["tree_sha"], + commit["url"], + ), + ) + + +# Mirrors the v3 server's `WRITE_CONFLICT_ATTEMPTS` +# (the `vortex-data/benchmarks-website` repo's `server/src/ingest.rs`). The DuckDB ingest wrapped +# `apply_envelope_once` in `retry_write_conflicts`; the DuckDB -> Postgres substrate change +# must preserve that behavior, since the dual-write CI runs ~14 concurrent writers. +_WRITE_CONFLICT_ATTEMPTS = 128 + + +def _retry_write_conflicts(op): + """Retry `op` on a Postgres write conflict, mirroring the v3 server's `retry_write_conflicts`. + + The v3 DuckDB ingest retried on write conflicts up to `WRITE_CONFLICT_ATTEMPTS` times; the + Postgres writer must preserve that, because the dual-write CI runs many concurrent writers + whose row-level `ON CONFLICT DO UPDATE` upserts touching the same `commits` / dim rows in + conflicting orders can deadlock. The retryable analogs on Postgres are deadlock + (`SQLSTATE 40P01`) and serialization failure (`40001`); both abort one transaction cleanly, + so re-running the whole transaction is safe. A non-retryable error (e.g. a validation + `SystemExit`) propagates immediately. Returns `op`'s value on the first success. + """ + from psycopg import errors as pg_errors + + for attempt in range(1, _WRITE_CONFLICT_ATTEMPTS + 1): + try: + return op() + except (pg_errors.DeadlockDetected, pg_errors.SerializationFailure): + # The failing `op`'s `with conn.transaction()` block already rolled back, so the + # connection is idle and the whole transaction can be retried. Re-raise on the + # final attempt (mirrors v3 returning the error after the last try; the bare + # retry loop mirrors v3's `std::thread::yield_now()` between attempts). + if attempt >= _WRITE_CONFLICT_ATTEMPTS: + raise + raise AssertionError("unreachable: _retry_write_conflicts exited without return or raise") + + +def ingest_postgres(conn, commit: dict, records: list[dict]) -> tuple[int, int]: + """Upsert a commit and its records into Postgres, retrying on write conflicts. + + Wraps `_ingest_postgres_once` in `_retry_write_conflicts` (mirroring the v3 server's + `apply_envelope = retry_write_conflicts(apply_envelope_once)`). Returns `(inserted, updated)` + aggregated across all fact tables. + """ + mid_mod = _measurement_id_module() + return _retry_write_conflicts(lambda: _ingest_postgres_once(conn, commit, records, mid_mod)) + + +def _ingest_postgres_once(conn, commit: dict, records: list[dict], mid_mod) -> tuple[int, int]: + """Upsert a commit and its records into Postgres in one transaction (a single attempt). + + Mirrors the v3 server's `apply_envelope_once`: upsert `commits` first, then each fact + record, classifying each as inserted or updated. Any validation failure rolls the whole + transaction back (all-or-nothing). Returns `(inserted, updated)` aggregated across all + fact tables. + """ + inserted = 0 + updated = 0 + with conn.transaction(): + _upsert_commit(conn, commit) + for idx, record in enumerate(records): + kind = _validate_record_fields(record, idx) + if record["commit_sha"] != commit["sha"]: + raise SystemExit( + f"record {idx} ({kind}): commit_sha {record['commit_sha']!r} does not " + f"match envelope commit.sha {commit['sha']!r}" + ) + _validate_record_values(record, kind, idx) + if _APPLY_RECORD[kind](conn, mid_mod, record): + updated += 1 + else: + inserted += 1 + return inserted, updated + + +def _region_from_host(host: str) -> str | None: + """Parse the AWS region out of an RDS endpoint hostname. + + RDS endpoints look like `...rds.amazonaws.com` (instance) + or `.proxy-..rds.amazonaws.com` (proxy); the region is the + label immediately before `rds.amazonaws.com`. Returns None for any other + shape. + """ + parts = host.split(".") + if len(parts) >= 4 and parts[-3:] == ["rds", "amazonaws", "com"]: + return parts[-4] + return None + + +def _rds_iam_token(*, host: str, port: int, user: str, region: str | None) -> str: + """Mint a short-lived RDS IAM auth token to use as the connection password.""" + import boto3 + + session = boto3.session.Session() + resolved = region or session.region_name or _region_from_host(host) + if not resolved: + raise SystemExit("could not determine the AWS region for the RDS IAM token; pass --region or set AWS_REGION.") + client = session.client("rds", region_name=resolved) + return client.generate_db_auth_token(DBHostname=host, Port=port, DBUsername=user, Region=resolved) + + +# The least-privilege login role the v4 CI ingest path must authenticate as +# (created by `migrations/004_ingest_role.sql`; SELECT,INSERT,UPDATE only). +# Phase-2 BAN: do not authenticate the ingest write path as `migrator` / +# `GitHubBenchmarkSchemaRole`. +_INGEST_ROLE = "bench_ingest" + + +def connect_postgres(dsn: str, region: str | None): + """Open a psycopg connection to the RDS Postgres ingest target. + + Enforces the ingest contract: verify-full TLS, and authentication only as the + least-privilege `bench_ingest` role -- always, regardless of auth method (IAM + token or password) or host. This is the production RDS ingest connector; the + test suite exercises ingest via `ingest_postgres` directly, so there is no + local-affordance exception to carve out. A host-based "is this local?" + heuristic would be unreliable anyway because libpq can resolve the host from a + DSN `hostaddr=` or the `$PGHOST` environment variable, either of which could + bypass a DSN-host check. Parses `dsn`; if it carries no password, mints an RDS + IAM auth token for the DSN's user and host (the DSN is expected to also supply + `sslrootcert` for verify-full to validate against). + """ + import psycopg + from psycopg import conninfo + + params = conninfo.conninfo_to_dict(dsn) + + # verify-full is the ingest TLS contract. Default an absent value, but + # refuse a DSN that explicitly downgrades it rather than silently weakening + # the internet-reachable ingest connection. + sslmode = params.get("sslmode") + if sslmode is None: + params["sslmode"] = "verify-full" + elif sslmode != "verify-full": + raise SystemExit( + f"--postgres requires sslmode=verify-full for the RDS ingest path; DSN " + f"specified sslmode={sslmode!r}. Omit it (defaults to verify-full) or set " + f"it to verify-full." + ) + + user = params.get("user") + # Least-privilege: always the bench_ingest role. No host heuristic (see the + # docstring): a misconfigured DSN -- by host=, hostaddr=, or $PGHOST -- must + # never ingest as migrator/postgres. + if user != _INGEST_ROLE: + raise SystemExit(f"--postgres must connect as the least-privilege {_INGEST_ROLE!r} role; DSN user is {user!r}.") + + if not params.get("password"): + # IAM-token path (production CI ingest): mint a token for the DSN's user. + host = params.get("host") + if not host: + raise SystemExit("--postgres DSN must specify host for IAM token minting") + try: + port = int(params.get("port", 5432)) + except (TypeError, ValueError) as exc: + raise SystemExit(f"--postgres DSN has a non-numeric port: {params.get('port')!r}") from exc + params["password"] = _rds_iam_token(host=host, port=port, user=user, region=region) + + # Force `search_path=public` so the writer's unqualified table names always resolve to the + # migration-owned `public.*` tables, regardless of any `search_path` baked into the DSN's + # `options=-c search_path=...` or the role's default. libpq applies repeated `-c` settings + # left-to-right (last wins), so appending ours last makes it authoritative even if the DSN + # already set one. + existing_options = params.get("options") or "" + params["options"] = f"{existing_options} -c search_path=public".strip() + + conn = psycopg.connect(**params) + # Verify the RESOLVED transport actually used TLS, not merely that the DSN requested + # verify-full. The `sslmode` check above rejects an explicit downgrade, but it cannot stop a + # hostless / Unix-socket DSN (host omitted, `host=/...`, or libpq resolving via `$PGHOST`) + # from connecting over a local socket, where libpq silently ignores `sslmode`. Checking + # `ssl_in_use` post-connect validates the connection that actually happened rather than + # trusting the DSN string -- the same "verify the resolved state" reasoning the docstring + # uses to reject a DSN-host heuristic -- and so also closes the `$PGHOST` / `hostaddr` bypass. + # NOTE: `ssl_in_use` lives on the low-level libpq wrapper `conn.pgconn` (a `pq.PGconn`); it is + # NOT on the high-level `conn.info` (`ConnectionInfo`), which exposes only host/dbname/user/etc. + if not conn.pgconn.ssl_in_use: + conn.close() + raise SystemExit( + "--postgres requires a verify-full TLS connection, but the established connection is " + "not using TLS (a hostless or Unix-socket DSN bypasses sslmode); connect to the RDS " + "instance over TCP with sslmode=verify-full." + ) + return conn + + +def _http(method: str, url: str, token: str | None, timeout: float) -> bytes: + """Issue one HTTP request and return the body. Raises on any non-2xx or + transport error; callers in `refresh_site_cache` swallow those.""" + headers = {"accept": "application/json"} + if token is not None: + headers["authorization"] = f"Bearer {token}" + req = urllib.request.Request(url, method=method, headers=headers) + with urllib.request.urlopen(req, timeout=timeout) as resp: + return resp.read() + + +def _warm_default_windows(base: str, timeout: float) -> None: + """Best-effort warm pass: prime the freshly invalidated Data Cache for the + landing page and every group's default last-100 bundle, so the first human + request after an ingest is already hot. Each request is independent; one + failure does not abort the others.""" + + def warm(url: str) -> None: + try: + _http("GET", url, None, timeout) + except Exception as exc: # noqa: BLE001 -- warm is best-effort. + print(f"warning: warm {url} failed: {exc}", file=sys.stderr) + + warm(f"{base}/") + try: + groups_body = _http("GET", f"{base}/api/groups", None, timeout) + slugs = [g["slug"] for g in json.loads(groups_body).get("groups", []) if "slug" in g] + except Exception as exc: # noqa: BLE001 -- group discovery is best-effort. + print(f"warning: warm group discovery failed: {exc}", file=sys.stderr) + return + # A whole-bundle recompute is a few seconds cold, so warm with bounded + # concurrency rather than one slow serial pass. + with ThreadPoolExecutor(max_workers=4) as pool: + pool.map(lambda s: warm(f"{base}/api/group/{s}?n=100"), slugs) + + +def refresh_site_cache(base_url: str, token: str, timeout: float) -> None: + """Revalidate the site's Data Cache tag, then warm the default windows. + + BEST-EFFORT: every failure is logged to stderr and swallowed so a cache + refresh can never change the ingest exit code. The warm pass is skipped + when revalidation fails: warming after a failed flush would repopulate the + Data Cache with stale data, which is the opposite of the intent. + """ + base = base_url.rstrip("/") + try: + _http("POST", f"{base}/api/revalidate", token, timeout) + except Exception as exc: # noqa: BLE001 -- refresh must never raise into ingest. + print(f"warning: cache revalidate failed: {exc}", file=sys.stderr) + return # Skip the warm pass: no point warming a cache that was not flushed. + _warm_default_windows(base, timeout) + + +def _main_postgres(args: argparse.Namespace) -> int: + records = read_records(args.jsonl_path) + # `build_commit` runs `git show `, so the SHA must be in the runner's local git + # history. The v4 ingest step inherits the v3 `--server` step's checkout assumption (the default + # checkout provides the head SHA); a shallow checkout missing the SHA fails loud here, and the + # v4 step is best-effort (continue-on-error), so it never fails the job. + commit = build_commit(args.commit_sha, args.repo_url, args.git_dir) + conn = connect_postgres(args.postgres, args.region) + try: + inserted, updated = ingest_postgres(conn, commit, records) + finally: + conn.close() + print( + json.dumps( + {"records": len(records), "inserted": inserted, "updated": updated}, + separators=(",", ":"), + ) + ) + # Best-effort site-cache refresh after a successful write. No-op unless both + # env vars are set (so the script stays inert until the ops wiring lands), + # and it can never fail the ingest. The ops prerequisite (setting the two env + # vars in Vercel and as GitHub secrets/vars) is documented in the "Ops + # prerequisite" section of .big-plans/ct__bench-v4-uiux-r3-design.md. + base_url = os.environ.get("BENCH_SITE_BASE_URL") + revalidate_token = os.environ.get("BENCH_REVALIDATE_TOKEN") + if base_url and revalidate_token: + refresh_site_cache(base_url, revalidate_token, args.timeout) + return 0 + + +def _main_server(args: argparse.Namespace) -> int: + if args.benchmark_id is None: + print("error: --benchmark-id is required in --server mode", file=sys.stderr) + return 2 token = os.environ.get(args.token_env) if not token: print( @@ -336,5 +1242,12 @@ def main() -> int: return 0 +def main() -> int: + args = parse_args() + if args.postgres is not None: + return _main_postgres(args) + return _main_server(args) + + if __name__ == "__main__": raise SystemExit(main()) From d6d8a8ea4f36761cd3a238d254888f9730ba9d78 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 11:59:38 -0400 Subject: [PATCH 16/33] scripts: add revalidate test + python-writer cross-check utility Port the pure-stdlib revalidate-hook test (asserts refresh_site_cache sends the bearer header and swallows every failure so it can never change the ingest exit code) and the cross_check_python_writer.py utility (confirms the Python writer recomputes the same measurement_id as seeded Rust-loaded rows and UPDATEs rather than duplicating) from the v4 emitter branch. Signed-off-by: Connor Tsui --- scripts/cross_check_python_writer.py | 247 ++++++++++++++++++++++ scripts/test_post_ingest_revalidate.py | 277 +++++++++++++++++++++++++ 2 files changed, 524 insertions(+) create mode 100644 scripts/cross_check_python_writer.py create mode 100644 scripts/test_post_ingest_revalidate.py diff --git a/scripts/cross_check_python_writer.py b/scripts/cross_check_python_writer.py new file mode 100644 index 00000000000..c2e582ed1a5 --- /dev/null +++ b/scripts/cross_check_python_writer.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors +"""Cross-check the Python ``--postgres`` ingest writer against seeded rows. + +The v4 historical seed (the PR-3.4 LOCAL rehearsal, then the PR-5.0 prod load) +loads the v3 DuckDB into Postgres via the Rust loader, which copies the +(Rust-computed) ``measurement_id`` values verbatim. This +harness confirms the LIVE property the upsert-not-duplicate invariant depends on: +the Python ``post-ingest.py --postgres`` writer, given a v3 envelope whose dim +tuple already exists in the seeded data, recomputes the SAME ``measurement_id`` +(Python port == Rust hash, golden-gated) and UPDATEs -- rather than +duplicate-INSERTs -- the seeded row, with the value columns round-tripping. + +Run it right after the seed (the PR-3.4 LOCAL rehearsal, then the PR-5.0 prod +seed) for earliest detection, and re-run it as the PR-5.1 pre-promotion gate: + + uv run scripts/cross_check_python_writer.py --postgres "$DSN" \\ + --envelopes real_envelopes.json + +``--envelopes`` is a JSON file ``{"commit": {...}, "records": [...]}`` in the v3 +envelope shape ``post-ingest.py --postgres`` consumes; every record's dim tuple +MUST already exist in the seeded data (so the writer UPDATEs, not INSERTs). The +harness exits non-zero on any problem: an INSERT where an UPDATE was expected (a +measurement_id that did not match a seeded row -> a would-be duplicate), or a +value column that did not round-trip. +""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import sys +from pathlib import Path + +_SCRIPTS_DIR = Path(__file__).resolve().parent + + +def _load_module(filename: str, modname: str): + """Load a sibling script by file path (the hyphen in some names blocks ``import``).""" + path = _SCRIPTS_DIR / filename + spec = importlib.util.spec_from_file_location(modname, path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +_TABLE_BY_KIND = { + "query_measurement": "query_measurements", + "compression_time": "compression_times", + "compression_size": "compression_sizes", + "random_access_time": "random_access_times", + "vector_search_run": "vector_search_runs", +} + +# Value (non-hashed) columns compared per kind, mirroring the verifier (PR-3.2) +# and the writer's ``ON CONFLICT DO UPDATE SET`` lists. Only the columns a record +# actually carries are compared; the rest default and are the writer's concern. +_VALUE_COLUMNS = { + "query_measurement": ( + "value_ns", + "all_runtimes_ns", + "peak_physical", + "peak_virtual", + "physical_delta", + "virtual_delta", + "env_triple", + ), + "compression_time": ("value_ns", "all_runtimes_ns", "env_triple"), + "compression_size": ("value_bytes",), + "random_access_time": ("value_ns", "all_runtimes_ns", "env_triple"), + "vector_search_run": ( + "value_ns", + "all_runtimes_ns", + "matches", + "rows_scanned", + "bytes_scanned", + "iterations", + "env_triple", + ), +} + + +def measurement_id_for(mid_mod, record: dict) -> int: + """Recompute a record's ``measurement_id`` independently of the writer.""" + kind = record["kind"] + if kind == "query_measurement": + return mid_mod.measurement_id_query( + commit_sha=record["commit_sha"], + dataset=record["dataset"], + dataset_variant=record.get("dataset_variant"), + scale_factor=record.get("scale_factor"), + query_idx=record["query_idx"], + storage=record["storage"], + engine=record["engine"], + format=record["format"], + ) + if kind == "compression_time": + return mid_mod.measurement_id_compression_time( + commit_sha=record["commit_sha"], + dataset=record["dataset"], + dataset_variant=record.get("dataset_variant"), + format=record["format"], + op=record["op"], + ) + if kind == "compression_size": + return mid_mod.measurement_id_compression_size( + commit_sha=record["commit_sha"], + dataset=record["dataset"], + dataset_variant=record.get("dataset_variant"), + format=record["format"], + ) + if kind == "random_access_time": + return mid_mod.measurement_id_random_access( + commit_sha=record["commit_sha"], + dataset=record["dataset"], + format=record["format"], + ) + if kind == "vector_search_run": + return mid_mod.measurement_id_vector_search( + commit_sha=record["commit_sha"], + dataset=record["dataset"], + layout=record["layout"], + flavor=record["flavor"], + threshold=record["threshold"], + ) + raise ValueError(f"unknown record kind {kind!r}") + + +def value_mismatches(db_row: dict, record: dict) -> list[str]: + """Value columns the record carries that disagree with the DB row. + + Only columns present in ``record`` are compared; the comparison is exact + (lists for ``all_runtimes_ns`` compare element-wise and order-sensitively). + """ + kind = record["kind"] + out: list[str] = [] + for col in _VALUE_COLUMNS[kind]: + if col not in record: + continue + expected = record[col] + actual = db_row.get(col) + if actual != expected: + out.append(f"{kind}.{col}: db={actual!r} != envelope={expected!r}") + return out + + +class CrossCheckReport: + """Outcome of one cross-check run.""" + + def __init__(self) -> None: + self.records = 0 + self.inserted = 0 + self.updated = 0 + self.problems: list[str] = [] + + def is_clean(self) -> bool: + """True iff every record UPDATEd a seeded row and all values round-tripped.""" + return self.inserted == 0 and not self.problems + + def __str__(self) -> str: + head = f"cross-check: {self.records} records, {self.updated} updated, {self.inserted} inserted" + if self.is_clean(): + return head + " -- CLEAN (writer UPDATEd seeded rows; values round-trip)" + return "\n".join([head + " -- FAILED", *(f" - {p}" for p in self.problems)]) + + +def cross_check(conn, commit: dict, records: list[dict]) -> CrossCheckReport: + """Run the writer over ``records`` (whose dim tuples must already be seeded) and + confirm it UPDATEs the seeded rows -- never duplicate-INSERTs -- with the value + columns round-tripping. + + ``conn`` is a live psycopg connection (the kind ``ingest_postgres`` accepts). + """ + post_ingest = _load_module("post-ingest.py", "post_ingest") + mid_mod = _load_module("_measurement_id.py", "_measurement_id") + + report = CrossCheckReport() + report.records = len(records) + expected = [(measurement_id_for(mid_mod, record), record) for record in records] + + inserted, updated = post_ingest.ingest_postgres(conn, commit, records) + report.inserted = inserted + report.updated = updated + if inserted: + report.problems.append( + f"{inserted} of {len(records)} records INSERTed where an UPDATE was expected: " + "a computed measurement_id did not match a seeded row, so the writer created a " + "DUPLICATE instead of upserting the seeded row" + ) + + for mid, record in expected: + table = _TABLE_BY_KIND[record["kind"]] + cols = [c for c in _VALUE_COLUMNS[record["kind"]] if c in record] + row = conn.execute( + f"SELECT {', '.join(cols)} FROM {table} WHERE measurement_id = %s", + (mid,), + ).fetchone() + if row is None: + report.problems.append(f"{record['kind']} measurement_id {mid}: no row found after ingest") + continue + report.problems.extend(value_mismatches(dict(zip(cols, row)), record)) + + return report + + +def _parse_args(argv=None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description="Cross-check the Python --postgres writer.") + parser.add_argument( + "--postgres", + required=True, + help="RDS DSN (verify-full TLS; connects as the bench_ingest role).", + ) + parser.add_argument( + "--region", + help="AWS region for RDS IAM token minting (when the DSN carries no password).", + ) + parser.add_argument( + "--envelopes", + required=True, + type=Path, + help='JSON file: {"commit": {...}, "records": [...]} in the v3 envelope shape.', + ) + return parser.parse_args(argv) + + +def main(argv=None) -> int: + args = _parse_args(argv) + post_ingest = _load_module("post-ingest.py", "post_ingest") + envelope = json.loads(args.envelopes.read_text()) + commit, records = envelope["commit"], envelope["records"] + if not records: + print("cross-check: envelope has no records", file=sys.stderr) + return 2 + conn = post_ingest.connect_postgres(args.postgres, args.region) + try: + report = cross_check(conn, commit, records) + finally: + conn.close() + print(report) + return 0 if report.is_clean() else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/test_post_ingest_revalidate.py b/scripts/test_post_ingest_revalidate.py new file mode 100644 index 00000000000..341de20efdf --- /dev/null +++ b/scripts/test_post_ingest_revalidate.py @@ -0,0 +1,277 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +"""Unit tests for post-ingest.py's best-effort site-cache refresh hook. + +These are pure-stdlib tests (no Docker, no psycopg): they exercise +`refresh_site_cache` by monkeypatching `urllib.request.urlopen`, asserting the +bearer header is sent and that every failure is swallowed so the hook can never +change the ingest exit code. +""" + +from __future__ import annotations + +import importlib.util +from pathlib import Path + +SCRIPTS_DIR = Path(__file__).resolve().parent + + +def _load_module(filename: str, modname: str): + path = SCRIPTS_DIR / filename + spec = importlib.util.spec_from_file_location(modname, path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +post_ingest = _load_module("post-ingest.py", "post_ingest") + + +class _FakeResponse: + def __init__(self, body: bytes = b"{}"): + self._body = body + + def read(self) -> bytes: + return self._body + + def __enter__(self): + return self + + def __exit__(self, *exc): + return False + + +def test_refresh_posts_revalidate_with_bearer(monkeypatch): + calls: list[tuple[str, dict[str, str], bytes | None]] = [] + + def fake_urlopen(req, timeout=None): + calls.append((req.full_url, dict(req.headers), req.data)) + return _FakeResponse(b'{"groups": []}') + + monkeypatch.setattr(post_ingest.urllib.request, "urlopen", fake_urlopen) + post_ingest.refresh_site_cache("https://example.test/", "tok", 5.0) + + revalidate = [c for c in calls if c[0].endswith("/api/revalidate")] + assert revalidate, "expected a POST to /api/revalidate" + # urllib title-cases header keys, so the bearer lives under "Authorization". + assert revalidate[0][1].get("Authorization") == "Bearer tok" + # Revalidate must be the first request issued, before any warm GETs. + assert calls[0][0].endswith("/api/revalidate"), "revalidate must precede warm GETs" + + +def test_refresh_skips_warm_when_revalidate_fails(monkeypatch): + """When the revalidate POST fails, no warm GET must be issued. + + Warming after a failed flush would repopulate the Data Cache with stale data. + The function must still return normally (never raise). + """ + calls: list[str] = [] + + def fake_urlopen(req, timeout=None): + calls.append(req.full_url) + if len(calls) == 1: + raise OSError("revalidate failed") + return _FakeResponse(b'{"groups": []}') + + monkeypatch.setattr(post_ingest.urllib.request, "urlopen", fake_urlopen) + result = post_ingest.refresh_site_cache("https://example.test", "tok", 5.0) + + # Must not raise. + assert result is None + # Only the one revalidate attempt should have been made; no warm GETs follow. + assert len(calls) == 1, f"expected 1 call (the failed revalidate), got {len(calls)}: {calls}" + assert calls[0].endswith("/api/revalidate") + + +def test_refresh_swallows_all_failures(monkeypatch): + def boom(req, timeout=None): + raise OSError("connection refused") + + monkeypatch.setattr(post_ingest.urllib.request, "urlopen", boom) + # Must not raise: a cache-refresh failure can never fail an ingest. + assert post_ingest.refresh_site_cache("https://example.test", "tok", 5.0) is None + + +def test_warm_default_windows_issues_expected_gets(monkeypatch): + """The warm pass GETs the landing page, /api/groups, and one /api/group/{slug}?n=100 + per discovered slug. + + Order is non-deterministic (ThreadPoolExecutor), so we assert the SET of + warm URLs rather than a fixed sequence. + """ + captured: list[str] = [] + + def fake_urlopen(req, timeout=None): + captured.append(req.full_url) + if req.full_url.endswith("/api/groups"): + body = b'{"groups":[{"slug":"g1"},{"slug":"g2"}]}' + else: + body = b"{}" + return _FakeResponse(body) + + monkeypatch.setattr(post_ingest.urllib.request, "urlopen", fake_urlopen) + post_ingest.refresh_site_cache("https://example.test/", "tok", 5.0) + + urls = set(captured) + # The revalidate POST is always first (tested separately); include it here + # so the assertion list is complete. + assert "https://example.test/api/revalidate" in urls + assert "https://example.test/" in urls + assert "https://example.test/api/groups" in urls + assert "https://example.test/api/group/g1?n=100" in urls + assert "https://example.test/api/group/g2?n=100" in urls + + +def test_main_postgres_refresh_failure_still_exits_zero(monkeypatch, tmp_path): + """When refresh_site_cache raises (or fails), _main_postgres still returns 0. + + This pins the "best-effort, never changes the exit code" contract: even if + the refresh throws, the successful write must be reported as exit code 0. + """ + import types + + # Provide the required env vars so the refresh branch is entered. + monkeypatch.setenv("BENCH_SITE_BASE_URL", "https://example.test") + monkeypatch.setenv("BENCH_REVALIDATE_TOKEN", "tok") + + # Make urlopen raise so refresh_site_cache encounters a failure. + def boom(req, timeout=None): + raise OSError("network error") + + monkeypatch.setattr(post_ingest.urllib.request, "urlopen", boom) + + # Stub out all DB/git dependencies so _main_postgres reaches the refresh + # call without needing a real Postgres connection or git history. + closed = {"n": 0} + conn = types.SimpleNamespace(close=lambda: closed.__setitem__("n", closed["n"] + 1)) + monkeypatch.setattr(post_ingest, "read_records", lambda path: []) + monkeypatch.setattr( + post_ingest, + "build_commit", + lambda *a, **k: { + "sha": "a" * 40, + "timestamp": "2026-01-02T03:04:05+00:00", + "message": "msg", + "author_name": "A", + "author_email": "a@example.com", + "committer_name": "A", + "committer_email": "a@example.com", + "tree_sha": "0" * 40, + "url": "https://example.com/commit/" + "a" * 40, + }, + ) + monkeypatch.setattr(post_ingest, "connect_postgres", lambda dsn, region: conn) + monkeypatch.setattr(post_ingest, "ingest_postgres", lambda c, commit, records: (0, 0)) + + import argparse + from pathlib import Path + + args = argparse.Namespace( + jsonl_path=Path("x.jsonl"), + commit_sha="a" * 40, + repo_url="https://example.com/repo", + git_dir=None, + postgres="dsn", + region=None, + timeout=5.0, + ) + rc = post_ingest._main_postgres(args) + # A refresh failure must never change the ingest exit code. + assert rc == 0 + assert closed["n"] == 1 # the connection was still closed in the finally block + + +def test_main_postgres_skips_refresh_when_base_url_absent(monkeypatch, tmp_path): + """When BENCH_SITE_BASE_URL is unset, refresh_site_cache is never called.""" + import types + + monkeypatch.delenv("BENCH_SITE_BASE_URL", raising=False) + monkeypatch.setenv("BENCH_REVALIDATE_TOKEN", "tok") + + refresh_calls: list[str] = [] + monkeypatch.setattr(post_ingest, "refresh_site_cache", lambda *a, **k: refresh_calls.append("called")) + + conn = types.SimpleNamespace(close=lambda: None) + monkeypatch.setattr(post_ingest, "read_records", lambda path: []) + monkeypatch.setattr( + post_ingest, + "build_commit", + lambda *a, **k: { + "sha": "b" * 40, + "timestamp": "2026-01-02T00:00:00+00:00", + "message": "m", + "author_name": "B", + "author_email": "b@example.com", + "committer_name": "B", + "committer_email": "b@example.com", + "tree_sha": "0" * 40, + "url": "https://example.com/commit/" + "b" * 40, + }, + ) + monkeypatch.setattr(post_ingest, "connect_postgres", lambda dsn, region: conn) + monkeypatch.setattr(post_ingest, "ingest_postgres", lambda c, commit, records: (0, 0)) + + import argparse + from pathlib import Path + + args = argparse.Namespace( + jsonl_path=Path("x.jsonl"), + commit_sha="b" * 40, + repo_url="https://example.com/repo", + git_dir=None, + postgres="dsn", + region=None, + timeout=5.0, + ) + rc = post_ingest._main_postgres(args) + assert rc == 0 + assert refresh_calls == [], "refresh_site_cache must not be called when BENCH_SITE_BASE_URL is absent" + + +def test_main_postgres_skips_refresh_when_token_absent(monkeypatch, tmp_path): + """When BENCH_REVALIDATE_TOKEN is unset, refresh_site_cache is never called.""" + import types + + monkeypatch.setenv("BENCH_SITE_BASE_URL", "https://example.test") + monkeypatch.delenv("BENCH_REVALIDATE_TOKEN", raising=False) + + refresh_calls: list[str] = [] + monkeypatch.setattr(post_ingest, "refresh_site_cache", lambda *a, **k: refresh_calls.append("called")) + + conn = types.SimpleNamespace(close=lambda: None) + monkeypatch.setattr(post_ingest, "read_records", lambda path: []) + monkeypatch.setattr( + post_ingest, + "build_commit", + lambda *a, **k: { + "sha": "c" * 40, + "timestamp": "2026-01-02T00:00:00+00:00", + "message": "m", + "author_name": "C", + "author_email": "c@example.com", + "committer_name": "C", + "committer_email": "c@example.com", + "tree_sha": "0" * 40, + "url": "https://example.com/commit/" + "c" * 40, + }, + ) + monkeypatch.setattr(post_ingest, "connect_postgres", lambda dsn, region: conn) + monkeypatch.setattr(post_ingest, "ingest_postgres", lambda c, commit, records: (0, 0)) + + import argparse + from pathlib import Path + + args = argparse.Namespace( + jsonl_path=Path("x.jsonl"), + commit_sha="c" * 40, + repo_url="https://example.com/repo", + git_dir=None, + postgres="dsn", + region=None, + timeout=5.0, + ) + rc = post_ingest._main_postgres(args) + assert rc == 0 + assert refresh_calls == [], "refresh_site_cache must not be called when BENCH_REVALIDATE_TOKEN is absent" From 3ef63fdc425c1c99cd290bb264f0657f62661e91 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:05:46 -0400 Subject: [PATCH 17/33] scripts: add adapted testcontainer writer test + self-contained schema fixture Port the testcontainer Postgres writer test and adapt it off the out-of-scope migrations/ + migrate-schema.py + benchmarks-website/server/src/schema.rs: apply a self-contained scripts/_v4_schema_fixture.sql (the 6-table DDL mirroring the website repo's migrations/001 plus the commit_timestamp column from 006 and its covering index from 007) and self-check SCHEMA_VERSION == 1. The suite spins postgres:16-alpine and exercises the real upsert/transaction path, the load-bearing pre-merge confidence gate with no prod or develop dependency. Signed-off-by: Connor Tsui --- scripts/_v4_schema_fixture.sql | 163 ++++ scripts/test_post_ingest_postgres.py | 1306 ++++++++++++++++++++++++++ 2 files changed, 1469 insertions(+) create mode 100644 scripts/_v4_schema_fixture.sql create mode 100644 scripts/test_post_ingest_postgres.py diff --git a/scripts/_v4_schema_fixture.sql b/scripts/_v4_schema_fixture.sql new file mode 100644 index 00000000000..e08ca9f5669 --- /dev/null +++ b/scripts/_v4_schema_fixture.sql @@ -0,0 +1,163 @@ +-- SPDX-License-Identifier: Apache-2.0 +-- SPDX-FileCopyrightText: Copyright the Vortex contributors + +-- TEST-ONLY fixture mirroring vortex-data/benchmarks-website migrations/001_initial_schema.sql +-- (plus the query_measurements.commit_timestamp column added by 006_read_path_perf.sql and the +-- covering index from 007_summary_covering_index.sql). Drift from the website repo is managed +-- by the SCHEMA_VERSION / column-list cross-repo contract; the role/grant migrations (002-005) +-- are omitted because the test suite connects as the container superuser (no roles needed). + +-- Dim table. `commit_sha` is the PK; `timestamp`, `tree_sha`, and `url` are +-- required (the server cannot render a chart without them); `message` and the +-- author/committer name + email pairs are nullable so v2-imported rows that +-- lacked them survive. +CREATE TABLE IF NOT EXISTS commits ( + commit_sha TEXT PRIMARY KEY NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + message TEXT, + author_name TEXT, + author_email TEXT, + committer_name TEXT, + committer_email TEXT, + tree_sha TEXT NOT NULL, + url TEXT NOT NULL +); + +-- Every chart resolves its x-axis from `commits` ordered by `timestamp`: the +-- window subquery takes the most recent `n` via `ORDER BY timestamp DESC, +-- commit_sha DESC LIMIT ?`, and the eligible-commits CTE orders ascending. A +-- single DESC index serves the LIMIT directly and is scanned backwards for the +-- ascending case. +CREATE INDEX IF NOT EXISTS idx_commits_timestamp + ON commits (timestamp DESC, commit_sha DESC); + +-- SQL query suite measurements. Natural key: `(commit_sha, dataset, +-- dataset_variant, scale_factor, query_idx, storage, engine, format)`. The four +-- memory columns are populated together or all NULL ("all four or none", +-- enforced by the ingest path, not by a DB constraint). `commit_timestamp` is a +-- denormalized copy of `commits.timestamp` (added by migration 006) used by the +-- latest-per-series summary index scan; nullable so existing rows survive the +-- migration without a backfill precondition. +CREATE TABLE IF NOT EXISTS query_measurements ( + measurement_id BIGINT PRIMARY KEY NOT NULL, + commit_sha TEXT NOT NULL, + dataset TEXT NOT NULL, + dataset_variant TEXT, + scale_factor TEXT, + query_idx INTEGER NOT NULL, + storage TEXT NOT NULL, + engine TEXT NOT NULL, + format TEXT NOT NULL, + value_ns BIGINT NOT NULL, + all_runtimes_ns BIGINT[] NOT NULL, + peak_physical BIGINT, + peak_virtual BIGINT, + physical_delta BIGINT, + virtual_delta BIGINT, + env_triple TEXT, + commit_timestamp TIMESTAMPTZ +); + +-- The chart query filters on `(dataset, dataset_variant, scale_factor, storage, +-- query_idx)` and joins to `commits`; `engine` and `format` are projected into +-- series tags, not filtered, so they are not part of the index. +CREATE INDEX IF NOT EXISTS idx_query_measurements_chart + ON query_measurements (dataset, dataset_variant, scale_factor, storage, query_idx); + +-- Covering index for the latest-per-series summary (migration 007): resolves +-- `DISTINCT ON (query_idx, engine, format) ORDER BY commit_timestamp DESC` to an +-- Index Only Scan. +CREATE INDEX IF NOT EXISTS idx_query_measurements_summary + ON query_measurements (dataset, dataset_variant, scale_factor, storage, + query_idx, engine, format, commit_timestamp DESC) + INCLUDE (value_ns); + +-- Low-cardinality indexes backing the loose-index scan in `collectFilterUniverse` +-- (migration 006). +CREATE INDEX IF NOT EXISTS idx_query_measurements_engine + ON query_measurements (engine); +CREATE INDEX IF NOT EXISTS idx_query_measurements_format + ON query_measurements (format); + +-- Encode/decode timings from `compress-bench`. Natural key: `(commit_sha, +-- dataset, dataset_variant, format, op)`, where `op IN ('encode', 'decode')`. +CREATE TABLE IF NOT EXISTS compression_times ( + measurement_id BIGINT PRIMARY KEY NOT NULL, + commit_sha TEXT NOT NULL, + dataset TEXT NOT NULL, + dataset_variant TEXT, + format TEXT NOT NULL, + op TEXT NOT NULL, + value_ns BIGINT NOT NULL, + all_runtimes_ns BIGINT[] NOT NULL, + env_triple TEXT +); + +CREATE INDEX IF NOT EXISTS idx_compression_times_chart + ON compression_times (dataset, dataset_variant); + +CREATE INDEX IF NOT EXISTS idx_compression_times_format + ON compression_times (format); + +-- On-disk sizes from `compress-bench`. One-shot (no per-iteration data, no +-- `all_runtimes_ns`). Natural key: `(commit_sha, dataset, dataset_variant, +-- format)`. Compression ratios are computed at query time, not stored. +CREATE TABLE IF NOT EXISTS compression_sizes ( + measurement_id BIGINT PRIMARY KEY NOT NULL, + commit_sha TEXT NOT NULL, + dataset TEXT NOT NULL, + dataset_variant TEXT, + format TEXT NOT NULL, + value_bytes BIGINT NOT NULL +); + +CREATE INDEX IF NOT EXISTS idx_compression_sizes_chart + ON compression_sizes (dataset, dataset_variant); + +CREATE INDEX IF NOT EXISTS idx_compression_sizes_format + ON compression_sizes (format); + +-- Take-time timings from `random-access-bench`. Natural key: `(commit_sha, +-- dataset, format)` -- no `dataset_variant`. Its own table so dataset filters +-- never have to disambiguate which suite a row belongs to. +CREATE TABLE IF NOT EXISTS random_access_times ( + measurement_id BIGINT PRIMARY KEY NOT NULL, + commit_sha TEXT NOT NULL, + dataset TEXT NOT NULL, + format TEXT NOT NULL, + value_ns BIGINT NOT NULL, + all_runtimes_ns BIGINT[] NOT NULL, + env_triple TEXT +); + +-- Single-column index: the random-access chart query filters on `dataset` only. +CREATE INDEX IF NOT EXISTS idx_random_access_times_chart + ON random_access_times (dataset); + +CREATE INDEX IF NOT EXISTS idx_random_access_times_format + ON random_access_times (format); + +-- Cosine-similarity scans from `vector-search-bench`. Emits a timing plus side +-- counters (`matches`, `rows_scanned`, `bytes_scanned`) for the same scan. +-- Natural key: `(commit_sha, dataset, layout, flavor, threshold)`. `iterations` +-- is a side count, not part of the dim hash. +CREATE TABLE IF NOT EXISTS vector_search_runs ( + measurement_id BIGINT PRIMARY KEY NOT NULL, + commit_sha TEXT NOT NULL, + dataset TEXT NOT NULL, + layout TEXT NOT NULL, + flavor TEXT NOT NULL, + threshold DOUBLE PRECISION NOT NULL, + value_ns BIGINT NOT NULL, + all_runtimes_ns BIGINT[] NOT NULL, + matches BIGINT NOT NULL, + rows_scanned BIGINT NOT NULL, + bytes_scanned BIGINT NOT NULL, + iterations INTEGER NOT NULL, + env_triple TEXT +); + +-- The chart query filters on `(dataset, layout, threshold)`; `flavor` is the +-- series key (projected, not filtered). +CREATE INDEX IF NOT EXISTS idx_vector_search_runs_chart + ON vector_search_runs (dataset, layout, threshold); diff --git a/scripts/test_post_ingest_postgres.py b/scripts/test_post_ingest_postgres.py new file mode 100644 index 00000000000..b5c1a08a3da --- /dev/null +++ b/scripts/test_post_ingest_postgres.py @@ -0,0 +1,1306 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +"""Integration + unit tests for `post-ingest.py`'s `--postgres` (v4) mode. + +The testcontainer tests apply the self-contained `_v4_schema_fixture.sql` (the +6-table DDL mirroring `vortex-data/benchmarks-website/migrations/001` plus the +`query_measurements.commit_timestamp` column from `006` and its covering index +from `007`) against a vanilla `postgres:16-alpine` and exercise +`post_ingest.ingest_postgres` against the resulting schema. They pin the +writer's load-bearing behavior-preservation invariants: + +- insert-vs-update accounting (re-ingesting an identical envelope upserts: 0 + inserted, N updated; row counts unchanged); +- `ON CONFLICT (measurement_id) DO UPDATE` overwrites value columns while + leaving the dim tuple (hence the `measurement_id`) untouched; +- every stored `measurement_id` matches the Python port in `_measurement_id.py`; +- a non-finite f64 dim (`vector_search_run.threshold`) raises loudly and writes + nothing (the transaction rolls back); +- the v3 server's `deny_unknown_fields`, `storage IN ('nvme','s3')`, + memory-quartet, and envelope-commit-sha validations are reproduced. + +The non-Docker unit tests cover `SCHEMA_VERSION` self-check, RDS hostname +region parsing, and the IAM-token-vs-DSN-password branch of `connect_postgres` +(boto3 + psycopg mocked). + +`post-ingest.py` is loaded by file path because the hyphen in its name blocks +`import`. +""" + +import importlib.util +import inspect +import json +import os +import re +import sys +import types +from collections.abc import Iterator +from pathlib import Path + +import psycopg +import pytest +from testcontainers.postgres import PostgresContainer + +SCRIPTS_DIR = Path(__file__).resolve().parent +_SCHEMA_FIXTURE = SCRIPTS_DIR / "_v4_schema_fixture.sql" + +COMMIT_SHA = "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0" + +_FACT_TABLES = [ + "query_measurements", + "compression_times", + "compression_sizes", + "random_access_times", + "vector_search_runs", +] + +_TABLE_BY_KIND = { + "query_measurement": "query_measurements", + "compression_time": "compression_times", + "compression_size": "compression_sizes", + "random_access_time": "random_access_times", + "vector_search_run": "vector_search_runs", +} + + +def _load_module(filename: str, modname: str): + """Load a sibling script by file path (the hyphen blocks `import`).""" + path = SCRIPTS_DIR / filename + spec = importlib.util.spec_from_file_location(modname, path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +post_ingest = _load_module("post-ingest.py", "post_ingest") +cross_check_mod = _load_module("cross_check_python_writer.py", "cross_check_python_writer") + + +def _sample_commit() -> dict: + return { + "sha": COMMIT_SHA, + "timestamp": "2026-01-02T03:04:05+00:00", + "message": "feat: a sample commit (#123)", + "author_name": "Ada Lovelace", + "author_email": "ada@example.com", + "committer_name": "Ada Lovelace", + "committer_email": "ada@example.com", + "tree_sha": "0123456789abcdef0123456789abcdef01234567", + "url": "https://github.com/vortex-data/vortex/commit/" + COMMIT_SHA, + } + + +def _sample_records() -> list[dict]: + """One record per fact-table kind, exercising every measurement_id path. + + `query_measurement` carries no memory quartet (all four NULL); the optional + string dims are a mix of present and absent to exercise `write_opt_str`. + """ + return [ + { + "kind": "query_measurement", + "commit_sha": COMMIT_SHA, + "dataset": "tpch", + "scale_factor": "1", + "query_idx": 7, + "storage": "nvme", + "engine": "vortex", + "format": "parquet", + "value_ns": 1230, + "all_runtimes_ns": [1230, 1300, 1180], + }, + { + "kind": "compression_time", + "commit_sha": COMMIT_SHA, + "dataset": "taxi", + "format": "vortex", + "op": "encode", + "value_ns": 500, + "all_runtimes_ns": [500, 520], + }, + { + "kind": "compression_size", + "commit_sha": COMMIT_SHA, + "dataset": "taxi", + "format": "vortex", + "value_bytes": 99999, + }, + { + "kind": "random_access_time", + "commit_sha": COMMIT_SHA, + "dataset": "chimp", + "format": "parquet", + "value_ns": 200, + "all_runtimes_ns": [200, 210], + }, + { + "kind": "vector_search_run", + "commit_sha": COMMIT_SHA, + "dataset": "cohere-large-10m", + "layout": "flat", + "flavor": "raw", + "threshold": 0.8, + "value_ns": 700, + "all_runtimes_ns": [700, 720], + "matches": 5, + "rows_scanned": 100, + "bytes_scanned": 2048, + "iterations": 3, + }, + ] + + +def _expected_mid(mid_mod, rec: dict) -> int: + """Recompute a record's measurement_id independently of the writer.""" + kind = rec["kind"] + if kind == "query_measurement": + return mid_mod.measurement_id_query( + commit_sha=rec["commit_sha"], + dataset=rec["dataset"], + dataset_variant=rec.get("dataset_variant"), + scale_factor=rec.get("scale_factor"), + query_idx=rec["query_idx"], + storage=rec["storage"], + engine=rec["engine"], + format=rec["format"], + ) + if kind == "compression_time": + return mid_mod.measurement_id_compression_time( + commit_sha=rec["commit_sha"], + dataset=rec["dataset"], + dataset_variant=rec.get("dataset_variant"), + format=rec["format"], + op=rec["op"], + ) + if kind == "compression_size": + return mid_mod.measurement_id_compression_size( + commit_sha=rec["commit_sha"], + dataset=rec["dataset"], + dataset_variant=rec.get("dataset_variant"), + format=rec["format"], + ) + if kind == "random_access_time": + return mid_mod.measurement_id_random_access( + commit_sha=rec["commit_sha"], + dataset=rec["dataset"], + format=rec["format"], + ) + if kind == "vector_search_run": + return mid_mod.measurement_id_vector_search( + commit_sha=rec["commit_sha"], + dataset=rec["dataset"], + layout=rec["layout"], + flavor=rec["flavor"], + threshold=rec["threshold"], + ) + raise AssertionError(f"unhandled kind {kind!r}") + + +def _count(conn: psycopg.Connection, table: str) -> int: + return conn.execute(f"SELECT count(*) FROM {table}").fetchone()[0] + + +def _docker_available() -> bool: + import subprocess + + try: + result = subprocess.run(["docker", "info"], capture_output=True, timeout=5, check=False) + return result.returncode == 0 + except (FileNotFoundError, subprocess.TimeoutExpired): + return False + + +def _require_docker_for_testcontainers() -> None: + # The testcontainer suite MUST run in CI: a missing Docker daemon is a HARD FAILURE there, + # not a silent skip. A green CI job with skipped testcontainer tests would let a Postgres-writer + # regression merge undetected, which is exactly what wiring `scripts/` into CI (PR-2.3) closes. + # Locally (no CI env) we skip so developers without Docker can still run the pure-unit tests. + if _docker_available(): + return + if os.environ.get("CI"): + pytest.fail("Docker unavailable in CI (`docker info` failed); the testcontainer suite must run, not skip") + pytest.skip("Docker not running; skipping Postgres testcontainer tests") + + +@pytest.fixture(scope="module") +def postgres_dsn() -> Iterator[str]: + """Module-scoped Postgres testcontainer; yields a libpq DSN. + + `BENCH_TEST_PG_DSN`, when set, is used verbatim instead of spinning up a + container (a superuser DSN against any throwaway Postgres). Otherwise a + `postgres:16-alpine` testcontainer is used, skipped locally when Docker is + unavailable but FAILED in CI (where the `CI` env var is set) so the + testcontainer suite can never silently skip on the CI runner. + """ + override = os.environ.get("BENCH_TEST_PG_DSN") + if override: + yield override + return + _require_docker_for_testcontainers() + with PostgresContainer("postgres:16-alpine") as container: + host = container.get_container_host_ip() + port = container.get_exposed_port(5432) + dsn = f"postgresql://{container.username}:{container.password}@{host}:{port}/{container.dbname}" + yield dsn + + +@pytest.fixture +def schema_conn(postgres_dsn: str) -> Iterator[psycopg.Connection]: + """A connection with the schema fixture freshly applied to empty tables. + + The container is module-scoped, so each test scrubs the `public` schema and + re-applies `_v4_schema_fixture.sql` (the `CREATE TABLE/INDEX IF NOT EXISTS` + statements are idempotent) to start from empty fact tables. + """ + with psycopg.connect(postgres_dsn) as conn: + conn.autocommit = True + # Guard the destructive scrub for the BENCH_TEST_PG_DSN OVERRIDE path only: + # that DSN is operator-supplied and honored verbatim, so refuse to drop the + # public schema unless the resolved host is loopback/socket. A fixture-spun + # testcontainer is throwaway by construction, so it is exempt -- and its + # get_container_host_ip() can be a non-loopback IP under TCP DOCKER_HOST / + # DinD / DooD, where this guard would otherwise fail the whole suite. + if os.environ.get("BENCH_TEST_PG_DSN"): + host = conn.info.host + if host and host not in {"localhost", "127.0.0.1", "::1"} and not host.startswith("/"): + pytest.fail( + f"schema_conn refuses to scrub a non-local Postgres host {host!r}; " + "point BENCH_TEST_PG_DSN at a loopback/socket throwaway database." + ) + with conn.cursor() as cur: + cur.execute( + """ + DO $$ + DECLARE r RECORD; + BEGIN + FOR r IN (SELECT tablename FROM pg_tables WHERE schemaname = 'public') LOOP + EXECUTE 'DROP TABLE IF EXISTS public.' || quote_ident(r.tablename) || ' CASCADE'; + END LOOP; + END$$; + """ + ) + cur.execute(_SCHEMA_FIXTURE.read_text()) + yield conn + + +def test_ingest_inserts_then_updates(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + records = _sample_records() + + inserted, updated = post_ingest.ingest_postgres(schema_conn, commit, records) + assert (inserted, updated) == (len(records), 0) + + # Re-ingesting the identical envelope upserts every row: 0 new, N updated. + inserted2, updated2 = post_ingest.ingest_postgres(schema_conn, commit, records) + assert (inserted2, updated2) == (0, len(records)) + + # One row per fact table, and exactly one commit row (no duplicates). + assert _count(schema_conn, "commits") == 1 + for kind, table in _TABLE_BY_KIND.items(): + assert _count(schema_conn, table) == 1, f"{kind} -> {table} row count drifted" + + +def test_query_measurement_insert_populates_commit_timestamp(schema_conn: psycopg.Connection) -> None: + """The denormalized `commit_timestamp` (migration 006, the read path's latest-per-series sort + key) is stamped from the envelope's `commits` row on BOTH upsert paths, so rows written by this + writer never depend on the post-deploy re-backfill.""" + commit = _sample_commit() + records = [r for r in _sample_records() if r["kind"] == "query_measurement"] + assert records, "expected at least one query_measurement sample record" + + def _unstamped_or_drifted() -> int: + # Count rows that are NOT stamped to their commit's timestamp -- 0 means every + # query_measurements row was correctly stamped (not just an arbitrary fetchone row). + return schema_conn.execute( + "SELECT count(*) FROM query_measurements q JOIN commits c USING (commit_sha)" + " WHERE q.commit_timestamp IS NULL OR q.commit_timestamp <> c.timestamp" + ).fetchone()[0] + + post_ingest.ingest_postgres(schema_conn, commit, records) + assert _count(schema_conn, "query_measurements") == len(records) + assert _unstamped_or_drifted() == 0 + + # The update path re-stamps as well: scrub the column, re-ingest the same envelope (an + # ON CONFLICT DO UPDATE), and every row's timestamp must come back. + schema_conn.execute("UPDATE query_measurements SET commit_timestamp = NULL") + post_ingest.ingest_postgres(schema_conn, commit, records) + assert _unstamped_or_drifted() == 0 + + +# Per-kind mutation of every value/side-counter/env column each table's ON +# CONFLICT DO UPDATE SET list owns (dim columns deliberately excluded). Mirrors +# the SET lists in benchmarks-website/server/src/ingest.rs; a stale/incorrect SET +# list in any table is caught by the parametrized update test below. +# `commit_timestamp` is also in the query_measurements SET list but is DERIVED +# from `commits` (not a record field), so it is pinned by +# `test_query_measurement_insert_populates_commit_timestamp` instead of here. +_UPDATE_VALUE_MUTATIONS = { + "query_measurement": { + "value_ns": 4242, + "all_runtimes_ns": [4242, 4243], + "peak_physical": 11, + "peak_virtual": 22, + "physical_delta": 3, + "virtual_delta": 4, + "env_triple": "aarch64-linux-gnu", + }, + "compression_time": {"value_ns": 4242, "all_runtimes_ns": [4242], "env_triple": "aarch64-linux-gnu"}, + "compression_size": {"value_bytes": 424242}, + "random_access_time": {"value_ns": 4242, "all_runtimes_ns": [4242], "env_triple": "aarch64-linux-gnu"}, + "vector_search_run": { + "value_ns": 4242, + "all_runtimes_ns": [4242], + "matches": 9, + "rows_scanned": 99, + "bytes_scanned": 999, + "iterations": 7, + "env_triple": "aarch64-linux-gnu", + }, +} + + +@pytest.mark.parametrize("kind", list(_UPDATE_VALUE_MUTATIONS)) +def test_update_overwrites_all_value_columns_per_table(schema_conn: psycopg.Connection, kind: str) -> None: + """For every fact table, re-ingesting the same dim tuple with changed value + columns updates in place: row count stays 1, measurement_id is unchanged, and + every column the ON CONFLICT SET list owns is overwritten. Guards all five SET + lists (not just query_measurements) against silent regression.""" + mid_mod = post_ingest._measurement_id_module() + table = _TABLE_BY_KIND[kind] + mutation = _UPDATE_VALUE_MUTATIONS[kind] + commit = _sample_commit() + rec = next(r for r in _sample_records() if r["kind"] == kind) + + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + mid = _expected_mid(mid_mod, rec) + + # Same dim tuple, every value/side-counter/env column changed -> must UPDATE. + mutated = dict(rec, **mutation) + inserted, updated = post_ingest.ingest_postgres(schema_conn, commit, [mutated]) + assert (inserted, updated) == (0, 1) + assert _count(schema_conn, table) == 1 + + cols = list(mutation) + row = schema_conn.execute(f"SELECT measurement_id, {', '.join(cols)} FROM {table}").fetchone() + assert row[0] == mid, f"{kind}: measurement_id changed on update (dim tuple should be stable)" + for i, col in enumerate(cols, start=1): + assert row[i] == mutation[col], f"{kind}.{col} was not overwritten by ON CONFLICT DO UPDATE" + + +def test_same_dim_tuple_twice_in_one_envelope_counts_second_as_update(schema_conn: psycopg.Connection) -> None: + """Two records with the same dim tuple in ONE envelope: the second is an + UPDATE, not a second insert, and last-write-wins. + + Pins the RETURNING (xmax = 0) classifier's same-transaction behavior + (empirically (1, 1), not (2, 0)): ON CONFLICT DO UPDATE locks the tuple the + first record created earlier in the same transaction, stamping its xmax with + the current txid, so the second is correctly classified as updated -- matching + the v3 server's exists()-preflight semantics. Resolves the cycle-2 reviewer + disagreement in favor of the empirical result.""" + commit = _sample_commit() + base = next(r for r in _sample_records() if r["kind"] == "compression_size") + second = dict(base, value_bytes=base["value_bytes"] + 1) # same dims, new value + + inserted, updated = post_ingest.ingest_postgres(schema_conn, commit, [base, second]) + assert (inserted, updated) == (1, 1) + assert _count(schema_conn, "compression_sizes") == 1 + stored = schema_conn.execute("SELECT value_bytes FROM compression_sizes").fetchone()[0] + assert stored == second["value_bytes"] # last-write-wins + + +def test_measurement_ids_match_python_port(schema_conn: psycopg.Connection) -> None: + mid_mod = post_ingest._measurement_id_module() + commit = _sample_commit() + records = _sample_records() + post_ingest.ingest_postgres(schema_conn, commit, records) + + for rec in records: + table = _TABLE_BY_KIND[rec["kind"]] + expected = _expected_mid(mid_mod, rec) + n = schema_conn.execute(f"SELECT count(*) FROM {table} WHERE measurement_id = %s", (expected,)).fetchone()[0] + assert n == 1, f"{rec['kind']}: no row stored under Python-computed measurement_id {expected}" + + +def test_empty_records_upserts_commit_only(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + inserted, updated = post_ingest.ingest_postgres(schema_conn, commit, []) + assert (inserted, updated) == (0, 0) + assert _count(schema_conn, "commits") == 1 + for table in _FACT_TABLES: + assert _count(schema_conn, table) == 0 + + +# The four fact tables that carry all_runtimes_ns (compression_sizes has none). +_ARRAY_TABLE_KINDS = ["query_measurement", "compression_time", "random_access_time", "vector_search_run"] + + +@pytest.mark.parametrize("kind", _ARRAY_TABLE_KINDS) +def test_empty_all_runtimes_ns_roundtrips(schema_conn: psycopg.Connection, kind: str) -> None: + # An empty all_runtimes_ns ([]) is a plausible producer output; psycopg sends + # it as the untyped literal '{}', which only types correctly because of the + # explicit ::bigint[] cast. Pin the cast for EVERY array-valued fact table so + # a dropped ::bigint[] in any of them fails loudly. + table = _TABLE_BY_KIND[kind] + commit = _sample_commit() + rec = next(r for r in _sample_records() if r["kind"] == kind) + rec = dict(rec, all_runtimes_ns=[]) + inserted, updated = post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert (inserted, updated) == (1, 0) + row = schema_conn.execute(f"SELECT all_runtimes_ns FROM {table}").fetchone() + assert row[0] == [] + + +@pytest.mark.parametrize("bad_runtimes", ["{}", [1, None], "not-a-list", [1, 2.5], [1, True]]) +def test_malformed_all_runtimes_ns_rejected(schema_conn: psycopg.Connection, bad_runtimes: object) -> None: + # all_runtimes_ns is presence-checked but the ::bigint[] cast is permissive: + # the string "{}" parses to an empty array and [1, null] adapts to {1,NULL}, + # both of which the v3 Vec serde rejects. Content validation must make + # these fail loud (record-indexed) and write nothing. + commit = _sample_commit() + rec = next(r for r in _sample_records() if r["kind"] == "compression_time") + rec = dict(rec, all_runtimes_ns=bad_runtimes) + with pytest.raises(SystemExit, match="array of integers"): + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert _count(schema_conn, "commits") == 0 + assert _count(schema_conn, "compression_times") == 0 + + +@pytest.mark.parametrize("bad", [float("nan"), float("inf"), float("-inf")]) +def test_nonfinite_threshold_raises_and_rolls_back(schema_conn: psycopg.Connection, bad: float) -> None: + commit = _sample_commit() + vsr = next(r for r in _sample_records() if r["kind"] == "vector_search_run") + vsr = dict(vsr, threshold=bad) + + with pytest.raises(SystemExit, match="not a finite number"): + post_ingest.ingest_postgres(schema_conn, commit, [vsr]) + + # All-or-nothing: the commit upsert that ran before the bad record is rolled + # back too, so nothing is written. + assert _count(schema_conn, "vector_search_runs") == 0 + assert _count(schema_conn, "commits") == 0 + + +# --- read_records / git_show_field unit tests (pure; no testcontainer needed) --- + + +def test_read_records_happy_path_mixed_newlines(tmp_path: Path) -> None: + # Pin read_records for valid input: records split across mixed line endings (LF, CRLF, and bare + # CR -- all handled by text-mode universal-newline translation), blank lines skipped, a + # multi-byte UTF-8 string round-tripped, and a final line with no trailing newline. + jsonl = tmp_path / "envelope.jsonl" + jsonl.write_bytes( + b'{"kind": "compression_size", "dataset": "a"}\n' # LF + b"\n" # blank line -> skipped + b'{"kind": "compression_size", "dataset": "b"}\r\n' # CRLF + b'{"kind": "compression_size", "dataset": "\xc3\xa9"}\r' # bare CR + UTF-8 e-acute + b'{"kind": "compression_size", "dataset": "d"}' # no trailing newline + ) + records = post_ingest.read_records(jsonl) + assert [r["dataset"] for r in records] == ["a", "b", "é", "d"] + + +def test_read_records_rejects_malformed_json(tmp_path: Path) -> None: + # A malformed JSON line fails loud with the path:line convention for debuggability. + jsonl = tmp_path / "envelope.jsonl" + jsonl.write_text('{"kind": "compression_size"\n', encoding="utf-8") + with pytest.raises(SystemExit, match=r"envelope\.jsonl:1: invalid JSON"): + post_ingest.read_records(jsonl) + + +def test_git_show_field_decodes_and_strips(monkeypatch) -> None: + # Well-formed multi-byte UTF-8 metadata round-trips through subprocess text mode and is stripped + # of the trailing newline git appends. + def fake_run(*_a, **kwargs): + assert kwargs.get("capture_output") is True, "git_show_field must capture stdout" + assert kwargs.get("text") is True, "git_show_field decodes git output as text" + return types.SimpleNamespace(stdout="café\n") + + monkeypatch.setattr(post_ingest.subprocess, "run", fake_run) + assert post_ingest.git_show_field("0" * 40, "%an", None) == "café" + + +def test_require_docker_fails_loud_in_ci(monkeypatch) -> None: + # In CI a missing Docker daemon must FAIL the testcontainer suite, not skip it: a green job with + # skipped Postgres tests would let a writer regression merge undetected (the gap PR-2.3 closes). + # Catch BOTH outcome types (Failed/Skipped both subclass BaseException) so an always-skip + # regression -- the helper raising Skipped -- is caught HERE and asserted against, rather than + # escaping a bare pytest.raises(Failed) and being recorded by pytest as a (green) test-skip. + monkeypatch.setenv("CI", "true") + monkeypatch.setattr(sys.modules[__name__], "_docker_available", lambda: False) + with pytest.raises((pytest.fail.Exception, pytest.skip.Exception)) as exc_info: + _require_docker_for_testcontainers() + assert isinstance(exc_info.value, pytest.fail.Exception), "must FAIL (not skip) when Docker is absent in CI" + + +def test_require_docker_skips_without_ci(monkeypatch) -> None: + # Locally (no CI env) a missing Docker daemon skips so the pure-unit tests still run. Symmetric to + # the CI test: catch both outcome types and assert it is a skip (not a fail). + monkeypatch.delenv("CI", raising=False) + monkeypatch.setattr(sys.modules[__name__], "_docker_available", lambda: False) + with pytest.raises((pytest.fail.Exception, pytest.skip.Exception)) as exc_info: + _require_docker_for_testcontainers() + assert isinstance(exc_info.value, pytest.skip.Exception), "must skip (not fail) when not in CI" + + +def test_retry_write_conflicts_retries_deadlock_then_succeeds() -> None: + # A deadlock aborts one writer's transaction; _retry_write_conflicts must re-run the whole + # transaction (mirroring the v3 server's retry_write_conflicts) until it succeeds, rather + # than surfacing the transient conflict to the caller. + calls = {"n": 0} + + def op() -> tuple[int, int]: + calls["n"] += 1 + if calls["n"] < 3: + raise psycopg.errors.DeadlockDetected("simulated deadlock") + return (4, 2) + + assert post_ingest._retry_write_conflicts(op) == (4, 2) + assert calls["n"] == 3 + + +def test_retry_write_conflicts_propagates_validation_error() -> None: + # A non-conflict error (e.g. a validation SystemExit) is deterministic and must propagate + # immediately, not be retried. + def op() -> tuple[int, int]: + raise SystemExit("record 0 (compression_size): bad") + + with pytest.raises(SystemExit, match="bad"): + post_ingest._retry_write_conflicts(op) + + +def test_retry_write_conflicts_gives_up_after_attempt_cap() -> None: + # A transaction that conflicts on every attempt eventually re-raises the conflict (loud) + # after exhausting _WRITE_CONFLICT_ATTEMPTS, rather than looping forever. + calls = {"n": 0} + + def op() -> tuple[int, int]: + calls["n"] += 1 + raise psycopg.errors.SerializationFailure("always conflicts") + + with pytest.raises(psycopg.errors.SerializationFailure): + post_ingest._retry_write_conflicts(op) + assert calls["n"] == post_ingest._WRITE_CONFLICT_ATTEMPTS + + +# Per fact-table _insert_ function: the dim columns that feed measurement_id and so MUST be +# EXCLUDED from the ON CONFLICT DO UPDATE SET clause (BAN: the upsert overwrites only value/env +# columns, never the dim tuple). commit_sha is intentionally NOT listed here: it is a dim-hash +# input but is included in the SET to match the v3 server exactly (a measurement_id conflict +# means the dims, incl. commit_sha, are identical, so the overwrite is a no-op). +_SET_EXCLUDED_DIM_COLUMNS = { + "_insert_query_measurement": [ + "dataset", + "dataset_variant", + "scale_factor", + "query_idx", + "storage", + "engine", + "format", + ], + "_insert_compression_time": ["dataset", "dataset_variant", "format", "op"], + "_insert_compression_size": ["dataset", "dataset_variant", "format"], + "_insert_random_access": ["dataset", "format"], + "_insert_vector_search": ["dataset", "layout", "flavor", "threshold"], +} + + +@pytest.mark.parametrize("fn_name,dim_columns", list(_SET_EXCLUDED_DIM_COLUMNS.items())) +def test_on_conflict_set_excludes_dim_columns(fn_name: str, dim_columns: list[str]) -> None: + # The existing per-table update test pins that value columns ARE in the SET list; this pins + # the inverse (load-bearing) BAN invariant: dim columns are NEVER in the SET list, so a future + # edit adding a dim to a DO UPDATE SET (which would overwrite the dim tuple on an upsert) fails + # loudly. Parse EVERY assigned (left-hand-side) column in each _insert_ function's SQL SET + # clause -- regardless of the right-hand side -- so a dim assigned from any RHS (e.g. + # `dataset = 'x'`, not just `dataset = excluded.dataset`) is also caught. + src = inspect.getsource(getattr(post_ingest, fn_name)) + match = re.search(r"DO UPDATE SET(.*?)RETURNING", src, re.DOTALL) + assert match, f"{fn_name}: no `DO UPDATE SET ... RETURNING` clause found" + set_columns = set(re.findall(r"(\w+)\s*=", match.group(1))) + assert set_columns, f"{fn_name}: parsed an empty SET column list (regex drift?)" + for dim in dim_columns: + assert dim not in set_columns, ( + f"{fn_name}: dim column {dim!r} must NOT be in the ON CONFLICT DO UPDATE SET clause " + f"(it feeds measurement_id); found SET columns: {sorted(set_columns)}" + ) + + +def test_unknown_field_rejected(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = dict(qm, surprise_field="x") + with pytest.raises(SystemExit, match="deny_unknown_fields"): + post_ingest.ingest_postgres(schema_conn, commit, [qm]) + assert _count(schema_conn, "commits") == 0 + + +def test_missing_required_field_rejected(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = {k: v for k, v in qm.items() if k != "engine"} + with pytest.raises(SystemExit, match="missing required field"): + post_ingest.ingest_postgres(schema_conn, commit, [qm]) + + +def test_unknown_kind_rejected(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + with pytest.raises(SystemExit, match="unknown kind"): + post_ingest.ingest_postgres(schema_conn, commit, [{"kind": "not_a_kind", "commit_sha": COMMIT_SHA}]) + + +@pytest.mark.parametrize("bad_kind", [[], {}, 5]) +def test_non_scalar_kind_rejected(schema_conn: psycopg.Connection, bad_kind: object) -> None: + # A non-string (unhashable) kind must hit the controlled record-indexed + # unknown-kind error, not a bare TypeError at the `in` membership check. + commit = _sample_commit() + with pytest.raises(SystemExit, match="unknown kind"): + post_ingest.ingest_postgres(schema_conn, commit, [{"kind": bad_kind, "commit_sha": COMMIT_SHA}]) + assert _count(schema_conn, "commits") == 0 + + +def test_validation_errors_carry_the_record_index(schema_conn: psycopg.Connection) -> None: + # A bad record at index 1 reports "record 1 (...)", matching the v3 server's + # indexed per-record errors. + commit = _sample_commit() + good = next(r for r in _sample_records() if r["kind"] == "compression_time") + bad = dict(next(r for r in _sample_records() if r["kind"] == "query_measurement"), storage="floppy") + with pytest.raises(SystemExit, match=r"record 1 \(query_measurement\): storage must be"): + post_ingest.ingest_postgres(schema_conn, commit, [good, bad]) + + +@pytest.mark.parametrize("bad", [[], "x", 5, None]) +def test_non_object_record_rejected(schema_conn: psycopg.Connection, bad: object) -> None: + # A JSONL line that parses to a non-object (list/str/int/null) must fail with + # the record-indexed SystemExit convention, not an uncontrolled AttributeError, + # and must write nothing (commit upsert rolls back). + commit = _sample_commit() + with pytest.raises(SystemExit, match="expected a JSON object"): + post_ingest.ingest_postgres(schema_conn, commit, [bad]) + assert _count(schema_conn, "commits") == 0 + + +def test_wrong_typed_scalar_fails_loud_and_writes_nothing(schema_conn: psycopg.Connection) -> None: + # A wrong-typed scalar integer field fails loud with a record-indexed error + # (via _validate_record_values -> _require_int) and rolls back, rather than + # writing a divergent row. + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = dict(qm, query_idx="not-an-int") + with pytest.raises(SystemExit, match="query_idx must be an integer"): + post_ingest.ingest_postgres(schema_conn, commit, [qm]) + assert _count(schema_conn, "commits") == 0 + assert _count(schema_conn, "query_measurements") == 0 + + +@pytest.mark.parametrize( + ("kind", "field", "bad"), + [ + ("query_measurement", "value_ns", 1230.5), + ("query_measurement", "peak_physical", 10.0), + ("compression_size", "value_bytes", 1.5), + ("vector_search_run", "matches", 5.0), + ("vector_search_run", "iterations", 3.5), + ("compression_time", "value_ns", True), + ], +) +def test_float_or_bool_scalar_value_rejected( + schema_conn: psycopg.Connection, kind: str, field: str, bad: object +) -> None: + # Integer value columns bind straight to BIGINT/INTEGER; psycopg adapts a + # Python float to float8 and Postgres assignment-casts (rounds) it -- where + # v3 serde i64/i32 rejects a float. _require_int must fail loud (record-indexed) + # and write nothing. (A memory-quartet field is set fully so only the type + # under test triggers the failure.) + commit = _sample_commit() + rec = next(r for r in _sample_records() if r["kind"] == kind) + if kind == "query_measurement" and field == "peak_physical": + rec = dict(rec, peak_physical=bad, peak_virtual=20, physical_delta=1, virtual_delta=2) + else: + rec = dict(rec, **{field: bad}) + with pytest.raises(SystemExit, match=f"{field} must be an integer"): + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert _count(schema_conn, "commits") == 0 + assert _count(schema_conn, _TABLE_BY_KIND[kind]) == 0 + + +@pytest.mark.parametrize( + ("kind", "field", "bad"), + [ + ("compression_time", "format", 123), # required str + ("vector_search_run", "layout", 5), # required str + ("query_measurement", "engine", ["x"]), # required str + ], +) +def test_required_string_field_type_rejected( + schema_conn: psycopg.Connection, kind: str, field: str, bad: object +) -> None: + # A non-string required field must fail loud (record-indexed) rather than + # reach a TEXT bind or crash in the hash encoder (.encode), matching v3 serde. + commit = _sample_commit() + rec = dict(next(r for r in _sample_records() if r["kind"] == kind), **{field: bad}) + with pytest.raises(SystemExit, match=f"{field} must be a string"): + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert _count(schema_conn, "commits") == 0 + + +@pytest.mark.parametrize( + ("kind", "field", "bad"), + [ + ("compression_time", "env_triple", 123), # Option + ("query_measurement", "dataset_variant", 1), # Option + ("query_measurement", "scale_factor", 2.0), # Option + ], +) +def test_optional_string_field_type_rejected( + schema_conn: psycopg.Connection, kind: str, field: str, bad: object +) -> None: + commit = _sample_commit() + rec = dict(next(r for r in _sample_records() if r["kind"] == kind), **{field: bad}) + with pytest.raises(SystemExit, match=f"{field} must be a string or null"): + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert _count(schema_conn, "commits") == 0 + + +@pytest.mark.parametrize( + ("kind", "field", "bad", "bits"), + [ + ("query_measurement", "query_idx", 2**31, 32), # i32 hash dim + ("vector_search_run", "iterations", 2**31, 32), # i32, NOT a hash dim + ("query_measurement", "value_ns", 2**63, 64), # i64 + ("compression_size", "value_bytes", -(2**63) - 1, 64), # i64 low end + ], +) +def test_int_out_of_range_rejected(schema_conn: psycopg.Connection, kind: str, field: str, bad: int, bits: int) -> None: + # An out-of-i32/i64 integer must fail loud (record-indexed) rather than later + # as an uncaught struct.error (i32 hash dims) or a raw Postgres 22003 overflow. + commit = _sample_commit() + rec = dict(next(r for r in _sample_records() if r["kind"] == kind), **{field: bad}) + with pytest.raises(SystemExit, match=f"out of int{bits} range"): + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert _count(schema_conn, "commits") == 0 + assert _count(schema_conn, _TABLE_BY_KIND[kind]) == 0 + + +def test_all_runtimes_element_out_of_i64_range_rejected(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + rec = dict(next(r for r in _sample_records() if r["kind"] == "compression_time"), all_runtimes_ns=[2**63]) + with pytest.raises(SystemExit, match="element out of int64 range"): + post_ingest.ingest_postgres(schema_conn, commit, [rec]) + assert _count(schema_conn, "commits") == 0 + + +def test_late_validation_failure_rolls_back_earlier_fact_row(schema_conn: psycopg.Connection) -> None: + # All-or-nothing: a valid record at index 0 writes a fact row inside the + # transaction, then an invalid record at index 1 fails validation -- the whole + # transaction (commit + the index-0 fact row) must roll back, not just abort + # the second record. + commit = _sample_commit() + good = next(r for r in _sample_records() if r["kind"] == "compression_time") + bad = dict(next(r for r in _sample_records() if r["kind"] == "compression_size"), value_bytes="not-an-int") + with pytest.raises(SystemExit, match=r"record 1 \(compression_size\): value_bytes must be an integer"): + post_ingest.ingest_postgres(schema_conn, commit, [good, bad]) + assert _count(schema_conn, "commits") == 0 + assert _count(schema_conn, "compression_times") == 0 # index-0 fact row rolled back + assert _count(schema_conn, "compression_sizes") == 0 + + +def test_storage_must_be_nvme_or_s3(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = dict(qm, storage="floppy") + with pytest.raises(SystemExit, match="storage must be"): + post_ingest.ingest_postgres(schema_conn, commit, [qm]) + + +def test_partial_memory_quartet_rejected(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = dict(qm, peak_physical=10) # only one of the four memory columns set + with pytest.raises(SystemExit, match="all four or none"): + post_ingest.ingest_postgres(schema_conn, commit, [qm]) + + +def test_full_memory_quartet_accepted(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = dict(qm, peak_physical=10, peak_virtual=20, physical_delta=1, virtual_delta=2) + inserted, updated = post_ingest.ingest_postgres(schema_conn, commit, [qm]) + assert (inserted, updated) == (1, 0) + row = schema_conn.execute( + "SELECT peak_physical, peak_virtual, physical_delta, virtual_delta FROM query_measurements" + ).fetchone() + assert row == (10, 20, 1, 2) + + +def test_commit_sha_mismatch_rejected(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + qm = next(r for r in _sample_records() if r["kind"] == "query_measurement") + qm = dict(qm, commit_sha="f" * 40) # does not match envelope commit.sha + with pytest.raises(SystemExit, match="does not match envelope"): + post_ingest.ingest_postgres(schema_conn, commit, [qm]) + + +# -- non-Docker unit tests -------------------------------------------------- + + +def test_schema_version_matches_rust_source() -> None: + """`post-ingest.py`'s `SCHEMA_VERSION` is self-checked to be 1. + + The cross-repo lockstep with `vortex-data/benchmarks-website` `server/src/schema.rs` is + enforced by the cross-repo CONTRACT.md, not testable from within this repo. This test + guards the constant against an accidental bump: a version change here is a coordinated + change that must land in both repos together. + """ + assert post_ingest.SCHEMA_VERSION == 1 + + +@pytest.mark.parametrize( + ("host", "expected"), + [ + ("vortex-bench-prod.abc123.us-east-1.rds.amazonaws.com", "us-east-1"), + ("vortex-bench-proxy.proxy-abc.us-west-2.rds.amazonaws.com", "us-west-2"), + ("localhost", None), + ("foo.bar.example.com", None), + ], +) +def test_region_from_host(host: str, expected: str | None) -> None: + assert post_ingest._region_from_host(host) == expected + + +def _fake_conn(ssl_in_use: bool = True): + """Stand-in psycopg connection for connect_postgres tests. + + connect_postgres reads only `.pgconn.ssl_in_use` (the post-connect TLS check) and calls + `.close()` (on the non-TLS rejection path) on the object psycopg.connect returns. The shape + mirrors the REAL psycopg API: `ssl_in_use` is on the low-level `conn.pgconn` (a `pq.PGconn`), + NOT on the high-level `conn.info` (`ConnectionInfo`); see test_pgconn_ssl_in_use_accessor. + """ + return types.SimpleNamespace(pgconn=types.SimpleNamespace(ssl_in_use=ssl_in_use), close=lambda: None) + + +def test_connect_postgres_mints_iam_token_when_passwordless(monkeypatch) -> None: + captured: dict = {} + + class FakeClient: + def generate_db_auth_token(self, **kwargs): + captured["token_args"] = kwargs + return "iam-token-xyz" + + class FakeSession: + region_name = None + + def client(self, service, region_name): + captured["client"] = (service, region_name) + return FakeClient() + + fake_boto3 = types.ModuleType("boto3") + fake_boto3.session = types.SimpleNamespace(Session=lambda: FakeSession()) + monkeypatch.setitem(sys.modules, "boto3", fake_boto3) + + def fake_connect(**params): + captured["connect_params"] = params + return _fake_conn() + + monkeypatch.setattr(psycopg, "connect", fake_connect) + + dsn = "postgresql://bench_ingest@db.abc.us-east-1.rds.amazonaws.com:5432/benchmarks?sslrootcert=/ca.pem" + result = post_ingest.connect_postgres(dsn, region=None) + + assert result.pgconn.ssl_in_use is True + assert captured["token_args"] == { + "DBHostname": "db.abc.us-east-1.rds.amazonaws.com", + "Port": 5432, + "DBUsername": "bench_ingest", + "Region": "us-east-1", + } + cp = captured["connect_params"] + assert cp["password"] == "iam-token-xyz" + assert cp["sslmode"] == "verify-full" + # The non-credential libpq params must flow through to psycopg.connect so the + # verify-full handshake validates against the right host + CA bundle. + assert cp["host"] == "db.abc.us-east-1.rds.amazonaws.com" + assert cp["dbname"] == "benchmarks" + assert cp["user"] == "bench_ingest" + assert cp["sslrootcert"] == "/ca.pem" + + +def test_connect_postgres_uses_dsn_password_without_token(monkeypatch) -> None: + # If boto3 were imported here it would fail, proving the token path is not + # taken when the DSN already carries a password. + monkeypatch.setitem(sys.modules, "boto3", None) + + captured: dict = {} + + def fake_connect(**params): + captured.update(params) + return _fake_conn() + + monkeypatch.setattr(psycopg, "connect", fake_connect) + + # A password-bearing DSN does not mint an IAM token. The user must be + # bench_ingest (always enforced); verify-full is still required. + dsn = "postgresql://bench_ingest:secret@localhost:5432/db?sslmode=verify-full" + post_ingest.connect_postgres(dsn, region="us-east-1") + + assert captured["password"] == "secret" + assert captured["sslmode"] == "verify-full" + + +@pytest.mark.parametrize( + "dsn", + [ + # Previously-bypassing shapes: loopback host, hostaddr-only (no host=, so the + # old DSN-host locality check saw None), and a real RDS host. All must be + # rejected now that bench_ingest is required unconditionally. + "postgresql://migrator:secret@localhost:5432/db?sslmode=verify-full", + "postgresql://migrator:secret@/db?hostaddr=203.0.113.10&sslmode=verify-full", + "postgresql://postgres:secret@db.abc.us-east-1.rds.amazonaws.com:5432/benchmarks?sslmode=verify-full", + ], +) +def test_connect_postgres_always_requires_bench_ingest_user(monkeypatch, dsn: str) -> None: + # The least-privilege check no longer relies on a (bypassable) host heuristic: + # any non-bench_ingest user is refused regardless of host / hostaddr / auth. + def boom(**params): + raise AssertionError("psycopg.connect must not be reached for a non-bench_ingest user") + + monkeypatch.setattr(psycopg, "connect", boom) + + with pytest.raises(SystemExit, match="bench_ingest"): + post_ingest.connect_postgres(dsn, region="us-east-1") + + +@pytest.mark.parametrize("weak", ["require", "disable", "prefer"]) +def test_connect_postgres_rejects_weak_sslmode(monkeypatch, weak: str) -> None: + # The sslmode contract is enforced before connect: a DSN that downgrades it + # must fail loudly, not silently weaken the ingest TLS posture. + def boom(**params): + raise AssertionError("psycopg.connect must not be reached for a weak sslmode") + + monkeypatch.setattr(psycopg, "connect", boom) + + dsn = f"postgresql://bench_ingest@db.abc.us-east-1.rds.amazonaws.com:5432/benchmarks?sslmode={weak}" + with pytest.raises(SystemExit, match="sslmode=verify-full"): + post_ingest.connect_postgres(dsn, region="us-east-1") + + +@pytest.mark.parametrize("bad_user", ["migrator", "postgres", "GitHubBenchmarkSchemaRole"]) +def test_connect_postgres_iam_path_rejects_non_bench_ingest_user(monkeypatch, bad_user: str) -> None: + # The IAM path mints a token for the DSN user; enforce least-privilege so a + # misconfigured DSN cannot ingest as migrator/postgres. boto3=None proves + # the token path is refused before it is reached. + monkeypatch.setitem(sys.modules, "boto3", None) + + def boom(**params): + raise AssertionError("psycopg.connect must not be reached for a non-bench_ingest IAM user") + + monkeypatch.setattr(psycopg, "connect", boom) + + dsn = f"postgresql://{bad_user}@db.abc.us-east-1.rds.amazonaws.com:5432/benchmarks" + with pytest.raises(SystemExit, match="bench_ingest"): + post_ingest.connect_postgres(dsn, region="us-east-1") + + +def test_connect_postgres_forces_public_search_path(monkeypatch) -> None: + # The writer uses unqualified table names, so connect_postgres pins search_path=public to + # defend against DSN/PGOPTIONS search_path drift. libpq applies repeated -c settings + # last-wins, so a hostile DSN search_path must be overridden by ours appended last. + captured: dict = {} + + def fake_connect(**params): + captured["connect_params"] = params + return _fake_conn() + + monkeypatch.setattr(psycopg, "connect", fake_connect) + + dsn = ( + "host=db.example.com port=5432 dbname=benchmarks user=bench_ingest " + "password=secret sslrootcert=/ca.pem options='-c search_path=evil'" + ) + post_ingest.connect_postgres(dsn, region=None) + options = captured["connect_params"]["options"] + assert options.rstrip().endswith("-c search_path=public"), options + + +def test_connect_postgres_rejects_non_tls_connection(monkeypatch) -> None: + # The verify-full contract must hold for the password branch too: a hostless / Unix-socket DSN + # bypasses sslmode (libpq does not TLS over a local socket), so connect_postgres asserts the + # RESOLVED connection actually used TLS (conn.pgconn.ssl_in_use) and fails loud + closes the + # connection otherwise -- the host check alone (IAM branch only) cannot catch this. + monkeypatch.setitem(sys.modules, "boto3", None) + closed = {"n": 0} + fake = types.SimpleNamespace( + pgconn=types.SimpleNamespace(ssl_in_use=False), + close=lambda: closed.__setitem__("n", closed["n"] + 1), + ) + monkeypatch.setattr(psycopg, "connect", lambda **params: fake) + dsn = "user=bench_ingest password=secret dbname=db sslmode=verify-full host=/var/run/postgresql" + with pytest.raises(SystemExit, match="not using TLS"): + post_ingest.connect_postgres(dsn, region=None) + assert closed["n"] == 1 # the rejected (non-TLS) connection is closed + + +def test_pgconn_ssl_in_use_accessor() -> None: + # Pin where ssl_in_use actually lives in psycopg so connect_postgres's post-connect TLS check + # cannot silently use the wrong accessor again (a prior cycle shipped conn.info.ssl_in_use, + # which raises AttributeError on every real connect): ssl_in_use is on the low-level pq.PGconn + # (reached via conn.pgconn), NOT on the high-level ConnectionInfo (conn.info). A psycopg upgrade + # that moves it trips this loudly, and it guarantees the _fake_conn mock shape matches reality. + assert hasattr(psycopg.pq.PGconn, "ssl_in_use") + assert not hasattr(psycopg.ConnectionInfo, "ssl_in_use") + + +def test_real_connection_exposes_pgconn_ssl_in_use(schema_conn: psycopg.Connection) -> None: + # Pin the conn.pgconn INTERMEDIATE traversal connect_postgres walks (not just the ssl_in_use + # leaf): a real psycopg connection must expose conn.pgconn.ssl_in_use, so a future psycopg + # rename of the .pgconn attribute trips loudly rather than being masked by the SimpleNamespace + # mocks the connect_postgres unit tests use (the cycle-9/10 regression class). + assert hasattr(schema_conn, "pgconn") + assert isinstance(schema_conn.pgconn.ssl_in_use, bool) + + +def test_main_postgres_composition(monkeypatch, capsys) -> None: + # Pin the production --postgres CLI wiring (no DB): read_records -> build_commit -> + # connect_postgres -> ingest_postgres -> compact JSON to stdout -> conn.close() in finally. + # Explicitly clear both refresh env vars so this test is robust to ambient env: + # the refresh branch is gated on both being set, and testing its skip is + # covered by test_post_ingest_revalidate.py. + monkeypatch.delenv("BENCH_SITE_BASE_URL", raising=False) + monkeypatch.delenv("BENCH_REVALIDATE_TOKEN", raising=False) + closed = {"n": 0} + conn = types.SimpleNamespace(close=lambda: closed.__setitem__("n", closed["n"] + 1)) + monkeypatch.setattr(post_ingest, "read_records", lambda path: [{"kind": "compression_size"}]) + monkeypatch.setattr(post_ingest, "build_commit", lambda *a, **k: _sample_commit()) + monkeypatch.setattr(post_ingest, "connect_postgres", lambda dsn, region: conn) + monkeypatch.setattr(post_ingest, "ingest_postgres", lambda c, commit, records: (3, 2)) + args = types.SimpleNamespace( + jsonl_path=Path("x.jsonl"), + commit_sha=COMMIT_SHA, + repo_url="https://example.com/repo", + git_dir=None, + postgres="dsn", + region=None, + timeout=30.0, + ) + rc = post_ingest._main_postgres(args) + assert rc == 0 + assert closed["n"] == 1 # conn closed in the finally block + assert json.loads(capsys.readouterr().out) == {"records": 1, "inserted": 3, "updated": 2} + + +# -- v3 --server path: argparse contract + dispatch (the refactor's new surface) -- + +_ARGV_BASE = ["post-ingest.py", "x.jsonl", "--commit-sha", "a" * 40, "--benchmark-id", "b"] + + +def test_server_and_postgres_are_mutually_exclusive(monkeypatch) -> None: + monkeypatch.setattr(sys, "argv", _ARGV_BASE + ["--server", "http://x", "--postgres", "postgresql://h/d"]) + with pytest.raises(SystemExit): + post_ingest.parse_args() + + +def test_exactly_one_mode_is_required(monkeypatch) -> None: + monkeypatch.setattr(sys, "argv", _ARGV_BASE) # neither --server nor --postgres + with pytest.raises(SystemExit): + post_ingest.parse_args() + + +def test_main_dispatches_to_server_vs_postgres(monkeypatch) -> None: + # The post-refactor main() routes by mode; pin that --server -> _main_server + # and --postgres -> _main_postgres without exercising the real ingest paths. + calls: list[str] = [] + monkeypatch.setattr(post_ingest, "_main_postgres", lambda args: calls.append("postgres") or 0) + monkeypatch.setattr(post_ingest, "_main_server", lambda args: calls.append("server") or 0) + + monkeypatch.setattr(sys, "argv", _ARGV_BASE + ["--postgres", "postgresql://bench_ingest@h:5432/d"]) + assert post_ingest.main() == 0 + monkeypatch.setattr(sys, "argv", _ARGV_BASE + ["--server", "http://x"]) + assert post_ingest.main() == 0 + + assert calls == ["postgres", "server"] + + +# Argv base without --benchmark-id, for the v4-optional / v3-required tests. +_ARGV_NO_BENCH_ID = ["post-ingest.py", "x.jsonl", "--commit-sha", "a" * 40] + + +def test_postgres_mode_does_not_require_benchmark_id(monkeypatch) -> None: + monkeypatch.setattr(sys, "argv", _ARGV_NO_BENCH_ID + ["--postgres", "postgresql://bench_ingest@h:5432/d"]) + args = post_ingest.parse_args() # must not raise -- --benchmark-id is v3-only + assert args.postgres == "postgresql://bench_ingest@h:5432/d" + assert args.benchmark_id is None + + +def test_server_mode_requires_benchmark_id(monkeypatch, capsys) -> None: + # parse_args accepts the omission (it's globally optional), but _main_server rejects it: + # --benchmark-id feeds the v3 envelope's run_meta. Set a valid token so the missing-benchmark-id + # check is the ONLY failing condition: _main_server checks benchmark_id BEFORE the token, so + # without a token a deleted benchmark-id check would still return 2 via the token path and pass + # for the wrong reason. Assert stderr names --benchmark-id to pin WHICH check rejected. + monkeypatch.setenv("INGEST_BEARER_TOKEN", "tok") + monkeypatch.setattr(sys, "argv", _ARGV_NO_BENCH_ID + ["--server", "http://x"]) + assert post_ingest.main() == 2 + assert "--benchmark-id" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- +# PR-3.5 cross-check harness: the Python --postgres writer UPDATEs (does not +# duplicate-INSERT) a pre-seeded row and the values round-trip. The local- +# container tests below validate the harness's discrimination; the PROD run +# (against the Rust-seeded RDS) is an operator gate. +# --------------------------------------------------------------------------- + + +def test_cross_check_clean_when_writer_updates_seeded_row(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + rec = _sample_records()[0] # query_measurement + mid = cross_check_mod.measurement_id_for(post_ingest._measurement_id_module(), rec) + # Seed the row DIRECTLY (independent of the writer -- a stand-in for a Rust- + # loaded row), with a deliberately-different value_ns/all_runtimes_ns so a clean + # cross-check proves the writer's UPDATE actually overwrote the seeded values. + schema_conn.execute( + """ + INSERT INTO query_measurements + (measurement_id, commit_sha, dataset, scale_factor, query_idx, storage, + engine, format, value_ns, all_runtimes_ns) + VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s::bigint[]) + """, + ( + mid, + rec["commit_sha"], + rec["dataset"], + rec["scale_factor"], + rec["query_idx"], + rec["storage"], + rec["engine"], + rec["format"], + 99, + [99], + ), + ) + + report = cross_check_mod.cross_check(schema_conn, commit, [rec]) + + assert report.is_clean(), str(report) + assert (report.inserted, report.updated) == (0, 1) + assert _count(schema_conn, "query_measurements") == 1 # UPDATE, not a duplicate + # The seeded value_ns (99) was overwritten by the envelope's value. + stored = schema_conn.execute( + "SELECT value_ns FROM query_measurements WHERE measurement_id = %s", (mid,) + ).fetchone()[0] + assert stored == rec["value_ns"] + + +def test_cross_check_clean_over_all_kinds(schema_conn: psycopg.Connection) -> None: + commit = _sample_commit() + records = _sample_records() # one record per fact-table kind + # Seed once (the stand-in for PR-3.4's prod seed), then the harness re-ingests: + # every record must UPDATE its seeded row, exercising all five measurement_id + # paths + value comparisons. + post_ingest.ingest_postgres(schema_conn, commit, records) + + report = cross_check_mod.cross_check(schema_conn, commit, records) + + assert report.is_clean(), str(report) + assert (report.inserted, report.updated) == (0, len(records)) + for table in _FACT_TABLES: + assert _count(schema_conn, table) == 1, f"{table} row count drifted" + + +def test_cross_check_flags_insert_when_row_not_seeded(schema_conn: psycopg.Connection) -> None: + # No pre-seed: the writer INSERTs (a duplicate, in prod terms) rather than + # UPDATEs, so the harness must FLAG it -- this is the discrimination that + # catches a Python-vs-Rust measurement_id divergence against the live seed. + commit = _sample_commit() + rec = _sample_records()[0] + + report = cross_check_mod.cross_check(schema_conn, commit, [rec]) + + assert not report.is_clean() + assert report.inserted == 1 + assert any("INSERT" in problem for problem in report.problems), report.problems + + +def test_cross_check_value_mismatches_discriminate() -> None: + rec = _sample_records()[0] # query_measurement: value_ns + all_runtimes_ns + matching = {"value_ns": rec["value_ns"], "all_runtimes_ns": rec["all_runtimes_ns"]} + assert cross_check_mod.value_mismatches(matching, rec) == [] + + wrong = {"value_ns": rec["value_ns"] + 1, "all_runtimes_ns": rec["all_runtimes_ns"]} + problems = cross_check_mod.value_mismatches(wrong, rec) + assert len(problems) == 1 and "value_ns" in problems[0] + + # all_runtimes_ns is compared element-wise and order-sensitively: a reorder mismatches. + reordered = {"value_ns": rec["value_ns"], "all_runtimes_ns": list(reversed(rec["all_runtimes_ns"]))} + assert any("all_runtimes_ns" in p for p in cross_check_mod.value_mismatches(reordered, rec)) + + # env_triple (omitted by the base sample) is discriminated when a record carries it. + rec_env = {**rec, "env_triple": "x86_64-linux"} + db_env = { + "value_ns": rec["value_ns"], + "all_runtimes_ns": rec["all_runtimes_ns"], + "env_triple": "aarch64-darwin", + } + assert any("env_triple" in p for p in cross_check_mod.value_mismatches(db_env, rec_env)) + + # vector_search side counters (matches/rows_scanned/bytes_scanned/iterations) discriminate. + vsr = _sample_records()[4] + counter_cols = ("value_ns", "all_runtimes_ns", "matches", "rows_scanned", "bytes_scanned", "iterations") + db_vsr = {c: vsr[c] for c in counter_cols} + db_vsr["matches"] = vsr["matches"] + 1 + assert any("matches" in p for p in cross_check_mod.value_mismatches(db_vsr, vsr)) + + +def test_cross_check_compares_env_and_memory_columns(schema_conn: psycopg.Connection) -> None: + # The base sample omits env_triple + the memory quartet, so the integration + # round-trip (writer UPDATE -> harness re-read -> compare) is otherwise never + # exercised on them. Seed a row whose env/memory/value DIFFER from the envelope, + # so a clean cross-check proves the writer UPDATEd those columns AND the harness + # compared them -- a writer SET-list omission of env_triple/memory would fail it. + commit = _sample_commit() + rec = { + **_sample_records()[0], + "env_triple": "aarch64-darwin", + "peak_physical": 11, + "peak_virtual": 22, + "physical_delta": 33, + "virtual_delta": 44, + } + mid = cross_check_mod.measurement_id_for(post_ingest._measurement_id_module(), rec) + schema_conn.execute( + """ + INSERT INTO query_measurements + (measurement_id, commit_sha, dataset, scale_factor, query_idx, storage, engine, + format, value_ns, all_runtimes_ns, peak_physical, peak_virtual, physical_delta, + virtual_delta, env_triple) + VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s::bigint[], %s, %s, %s, %s, %s) + """, + ( + mid, + rec["commit_sha"], + rec["dataset"], + rec["scale_factor"], + rec["query_idx"], + rec["storage"], + rec["engine"], + rec["format"], + 1, + [1], + 1, + 1, + 1, + 1, + "x86_64-linux", + ), + ) + + report = cross_check_mod.cross_check(schema_conn, commit, [rec]) + + assert report.is_clean(), str(report) + assert (report.inserted, report.updated) == (0, 1) + # The seeded env_triple + a memory column were overwritten by the envelope's values. + row = schema_conn.execute( + "SELECT env_triple, peak_physical FROM query_measurements WHERE measurement_id = %s", + (mid,), + ).fetchone() + assert row == ("aarch64-darwin", 11) From 96418dd527ac26090f748c9a5af4a9d5ff39c0ab Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:11:12 -0400 Subject: [PATCH 18/33] =?UTF-8?q?plan:=20set=20status=20reviewing=20?= =?UTF-8?q?=E2=80=94=20sub-phase=201.2=20gauntlet=20checkpoint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index e1f8d462797..e3b782037e5 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -107,7 +107,7 @@ v2/v3 paths. phase: "1: D — port v4 emitter (CODE)" sub_phase: "1.2 Postgres writer" task: null -status: implementing +status: reviewing last_gate: null phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` From 23dd35581b2fa3b80278e0379dcc43520797c6d7 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:24:29 -0400 Subject: [PATCH 19/33] plan: sub-phase 1.2 ledger entry + deferred carry-forward Record sub-phase 1.2 (Postgres writer) as gauntlet pr-3 accepted (1 cycle; fresh + correctness + maint, zero must-fix; 100 testcontainer tests green), and park the should-fix + nit findings in Deferred work (cross_check thin-commit hardening, real-PG retry coverage, fixture/CONTRACT.md doc accuracy, and minor port nits). Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 32 +++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index e3b782037e5..65f5a82c540 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -229,6 +229,28 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles - **Sub-phase 1.1**, `scripts/test_measurement_id.py` (`ensure_ascii`, 100-col), **nit**: add a comment explaining `ensure_ascii=False` is load-bearing for the multibyte guard; 3 lines exceed the 100-col style target (pass ruff's 120). Deferral rationale: cosmetic; optional cleanup in 1.3. +- **Sub-phase 1.2**, `scripts/cross_check_python_writer.py` -> `post-ingest.py:_upsert_commit`, + **should-fix**: the cross-check harness raises a bare `KeyError` on an operator-supplied commit + envelope missing an optional field (e.g. `message`). Deferral rationale: cross_check is operator + tooling (the future ingest-cutover gate), not the production write path (which always supplies all + 9 commit fields via `build_commit`); harden with `commit.get(...)` when next exercised. +- **Sub-phase 1.2**, `scripts/post-ingest.py` write-conflict retry, **should-fix**: the + deadlock/serialization retry is only unit-mocked (synthetic exceptions); no real-Postgres + abort+transaction-re-entry test. Deferral rationale: the retry is a verbatim port of the v3 + server's tested logic; a real-conflict testcontainer test is valuable but non-blocking for the + additive best-effort writer. +- **Sub-phase 1.2**, `scripts/_v4_schema_fixture.sql` header + `test_post_ingest_postgres.py` + docstrings, **should-fix** (maint): the fixture/test docs under-describe migration 006 (omit the + 5 format/engine indexes, which ARE present in the fixture), reference `CONTRACT.md` without its + `vortex-data/benchmarks-website/` path, and the "SCHEMA_VERSION / column-list contract" phrase + implies an automated drift guard that does not exist. Deferral rationale: docs-only; the fixture + CODE is column-by-column correct (opus-verified). Apply as a doc cleanup during phase-D + finalization (before the phase-end gauntlet). +- **Sub-phase 1.2**, multiple files, **nits** (~10): test rollback-assertion consistency + + revalidate HTTP-method assertion (fresh); non-string `commit_sha` yields a misleading mismatch + error (correctness); plan-cycle/PR labels in comments, `_load_module` 4x duplication, handler + name vs kind inconsistency, v3-path removal marker (maint). Deferral rationale: minor polish + carried from the verbatim port; not blocking. --- @@ -245,3 +267,13 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles extracted `vortex-data/benchmarks-website` repo; ruff-formatted to the repo's 120-col. - **Gauntlet:** pr-2 / accepted (cycles: 1) - **Deferred:** 5 items (see Carry-forward > Deferred work) + +#### Sub-phase 1.2: Postgres writer + +- **Shipped:** v4 `--postgres` IAM-auth upsert writer in `post-ingest.py` (verify-full TLS, + NaN/Inf guard, 5-table + commit-dim upsert in one transaction, write-conflict retry, best-effort + revalidate), v3 `--server` path kept stdlib-only; revalidate test + cross-check utility; adapted + testcontainer test + self-contained `_v4_schema_fixture.sql`. 100 testcontainer tests pass vs + postgres:16-alpine. +- **Gauntlet:** pr-3 / accepted (cycles: 1) — fresh + correctness (opus) + maint, zero must-fix. +- **Deferred:** 4 should-fix-class + ~10 nits (see Carry-forward > Deferred work) From dd4e812acd6e1680b6168f8fc43cc4d082eff664 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:24:55 -0400 Subject: [PATCH 20/33] plan: advance to sub-phase 1.3 (CI + workflow wiring) Sub-phase 1.2 complete + gauntlet-accepted; advance position to 1.3, status implementing. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 65f5a82c540..9fbce346f34 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -105,9 +105,9 @@ v2/v3 paths. ```yaml phase: "1: D — port v4 emitter (CODE)" -sub_phase: "1.2 Postgres writer" +sub_phase: "1.3 CI + workflow wiring" task: null -status: reviewing +status: implementing last_gate: null phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` From e4957e7b278d9dd4ef86f4ec370642c41a57c48f Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:29:23 -0400 Subject: [PATCH 21/33] plan: task-plan for sub-phase 1.3 (CI + workflow wiring) JIT writing-plans output for sub-phase 1.3: wire the scripts/ tests into ci.yml (contract+revalidate in python-test, testcontainer in a docker-gated job, all via uv run --with, no pyproject/uv.lock changes), and add the dormant best-effort v4 step to the three emitter workflows. 2 tasks. Signed-off-by: Connor Tsui --- ...4-emitters--1-3-ci-workflow-wiring.plan.md | 179 ++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 .big-plans/ct__bench-v4-emitters--1-3-ci-workflow-wiring.plan.md diff --git a/.big-plans/ct__bench-v4-emitters--1-3-ci-workflow-wiring.plan.md b/.big-plans/ct__bench-v4-emitters--1-3-ci-workflow-wiring.plan.md new file mode 100644 index 00000000000..51801b0eaeb --- /dev/null +++ b/.big-plans/ct__bench-v4-emitters--1-3-ci-workflow-wiring.plan.md @@ -0,0 +1,179 @@ +# Sub-phase 1.3 — CI + workflow wiring Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Wire the new scripts/ tests into CI and add the best-effort v4 ingest step block to the three emitter workflows, leaving the v4 path DORMANT (gated on `vars.GH_BENCH_INGEST_ROLE_ARN != ''`). + +**Architecture:** The v4 step blocks are ported verbatim from `f9b36ae3f` (already grill-me-correct: global RDS CA bundle, `uv run --no-project --with` deps, `continue-on-error` + env-gate). They insert at unchanged anchors on current develop (the v3 ingest step). The ci.yml test wiring is ADAPTED to our `--with` deps approach (NOT the branch's `--all-packages` + pyproject-deps approach): the cheap contract+revalidate tests run as a step in the existing required `python-test` job (no docker); the testcontainer test runs in a new lightweight docker-gated job. NO `pyproject.toml` / `uv.lock` changes (deps are supplied per-invocation via `--with`). + +**Tech Stack:** GitHub Actions YAML, yamllint, uv (`uv run --no-project --with`), Docker (testcontainer job). + +## Global Constraints + +- All `.github/` edits MUST pass `uvx yamllint --strict -c .yamllint.yaml ` (double-quote when quoting, 1-space `{ }`, 0-space `[ ]`, >=2-space inline comments, trailing newline, no trailing spaces). +- All `uses:` action references MUST be SHA-pinned with a `# vN` comment. Reuse SHAs already present in the repo where possible. +- Every v4 step MUST keep `if: vars.GH_BENCH_INGEST_ROLE_ARN != ''` AND `continue-on-error: true` (the dormancy + best-effort safety property). Do NOT weaken or remove the existing v3 `--server` steps, and do NOT add `continue-on-error` to them. +- NO changes to `pyproject.toml` or `uv.lock` (deps via `--with`). Do NOT add the v4 deps to the workspace. +- This sub-phase touches ONLY: `.github/workflows/ci.yml`, `.github/workflows/bench.yml`, `.github/workflows/sql-benchmarks.yml`, `.github/workflows/v3-commit-metadata.yml`. Do NOT touch any scripts/ file (those are 1.1/1.2) or post-ingest.py. +- Commits: `git commit -F` with a heredoc (NEVER backticks or a `---` line); sign off `Signed-off-by: Connor Tsui `. + +--- + +### Task 1: Wire the scripts/ tests into ci.yml + +**Files:** +- Modify: `.github/workflows/ci.yml` (add one step to the `python-test` job; add a new `scripts-test` job) + +- [ ] **Step 1: Add the contract + revalidate test step to the existing `python-test` job** + +In `.github/workflows/ci.yml`, the `python-test` job (`name: "Python (test)"`, ~line 75) has a `Pytest - Vortex` step (~line 93) with `working-directory: vortex-python/`. AFTER that step's block (and any sibling step in the job, e.g. a basedpyright step at ~line 104, i.e. as the LAST step of the `python-test` job), add a new step that runs from the repo root (no `working-directory`): + +```yaml + - name: "Pytest - scripts (measurement_id + revalidate)" + run: >- + uv run --no-project --with pytest --with xxhash pytest + scripts/test_measurement_id.py scripts/test_post_ingest_revalidate.py +``` + +This runs the two non-docker tests in the already-required `python-test` job (so they are required checks). It needs only `uv` (provided by the job's existing setup) + `xxhash` (via `--with`). Do NOT use `working-directory` (these paths are repo-root-relative). Match the surrounding indentation exactly. + +- [ ] **Step 2: Add a new docker-gated `scripts-test` job for the testcontainer test** + +Insert a new top-level job (2-space indent) into `.github/workflows/ci.yml` immediately BEFORE the `rust-docs:` job (~line 116). Copy the `actions/checkout@ # v6` line VERBATIM from an existing step in this same ci.yml (so the pinned SHA matches the repo); use the setup-uv action SHA `spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6` (the same SHA this PR introduces in the emitter workflows, Task 2): + +```yaml + scripts-test: + name: "Python (scripts testcontainers)" + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@ + - uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + # The testcontainer suite MUST run in CI, not silently skip: a green job with + # skipped tests would let a writer regression merge undetected. `docker info` + # fails the job up front if the daemon is unavailable, and the CI env var makes + # the testcontainer fixtures fail (not skip) per their _require_docker_for_testcontainers. + - name: "Verify Docker is available for testcontainers" + run: docker info + - name: "Pytest - scripts testcontainers" + run: >- + uv run --no-project --with pytest --with "psycopg[binary]" --with boto3 + --with xxhash --with testcontainers pytest scripts/test_post_ingest_postgres.py +``` + +Note: this is a LIGHTWEIGHT job (ubuntu-latest + uv + docker), NOT the branch's `--all-packages` + setup-prebuild + large-runner version -- our tests need no Rust-workspace build. It names the explicit test file (NOT `pytest scripts/`) so it does not collect the pandas-dependent `scripts/tests/test_benchmark_reporting.py` or the contract/revalidate tests already covered in Step 1. + +- [ ] **Step 3: Lint ci.yml** + +```bash +uvx yamllint --strict -c .yamllint.yaml .github/workflows/ci.yml +``` + +Expected: exit 0 (no diagnostics). Fix any yamllint violations (indentation, quoting, comment spacing, trailing newline) and re-run. + +- [ ] **Step 4: Confirm the wired commands actually pass locally (the exact commands CI will run)** + +```bash +uv run --no-project --with pytest --with xxhash pytest scripts/test_measurement_id.py scripts/test_post_ingest_revalidate.py -q +docker info >/dev/null 2>&1 && echo "docker OK" || echo "DOCKER MISSING -- escalate" +uv run --no-project --with pytest --with "psycopg[binary]" --with boto3 --with xxhash --with testcontainers pytest scripts/test_post_ingest_postgres.py -q +``` + +Expected: the first command passes (measurement_id 65 + revalidate 7); the testcontainer command passes (100 tests) under docker. These are the literal commands the two CI steps run, so green here means green in CI. + +- [ ] **Step 5: Commit** + +```bash +git add .github/workflows/ci.yml +git commit -F - <<'EOF' +ci: run the v4 emitter scripts/ tests + +Add the measurement_id contract test + the revalidate test to the python-test +job (required, no docker), and a lightweight docker-gated scripts-test job for +the testcontainer Postgres writer suite. Deps are supplied per-invocation via +uv run --with (no pyproject/uv.lock changes). + +Signed-off-by: Connor Tsui +EOF +``` + +--- + +### Task 2: Add the best-effort v4 ingest step to the three emitter workflows + +**Files:** +- Modify: `.github/workflows/bench.yml` +- Modify: `.github/workflows/sql-benchmarks.yml` +- Modify: `.github/workflows/v3-commit-metadata.yml` + +**Interfaces:** the v4 step runs `scripts/post-ingest.py --postgres` (sub-phase 1.2). The blocks are verbatim from `f9b36ae3f` and already correct. + +- [ ] **Step 1: Extract the three branch files for reference** + +```bash +git show f9b36ae3f:.github/workflows/bench.yml > /tmp/branch-bench.yml +git show f9b36ae3f:.github/workflows/sql-benchmarks.yml > /tmp/branch-sql.yml +git show f9b36ae3f:.github/workflows/v3-commit-metadata.yml > /tmp/branch-v3meta.yml +``` + +- [ ] **Step 2: Insert the v4 block into `bench.yml`** + +In `.github/workflows/bench.yml`, the `Ingest results to v3 server` step ends at the `--repo-url "${{ github.server_url }}/${{ github.repository }}"` line (~line 129), followed by a blank line and `- name: Alert incident.io` (~line 131). From `/tmp/branch-bench.yml`, copy the v4 block (the three steps `Configure AWS credentials for v4 ingest (OIDC)` / `Install uv for v4 ingest` / `Ingest results to v4 Postgres (best-effort)` plus the leading comment block) VERBATIM and insert it between the v3 step and `- name: Alert incident.io`, preserving the blank-line spacing and indentation. Do NOT alter the v3 step. The block's gate is `if: vars.GH_BENCH_INGEST_ROLE_ARN != ''` + `continue-on-error: true` on every step. + +- [ ] **Step 3: Insert the v4 block into `sql-benchmarks.yml`** + +Same as Step 2, but in `.github/workflows/sql-benchmarks.yml` (v3 step `--repo-url` at ~line 683, `- name: Alert incident.io` at ~line 685). Use the block from `/tmp/branch-sql.yml`. NOTE: this block's gate additionally carries `inputs.mode == 'develop' &&` on each `if:` (matching the v3 step's mode guard) -- keep that verbatim. + +- [ ] **Step 4: Insert the v4 block + `id-token` permission into `v3-commit-metadata.yml`** + +In `.github/workflows/v3-commit-metadata.yml`: +1. The `permissions:` block (~line 11) currently is `contents: read` only. Add `id-token: write` ABOVE `contents: read` (from `/tmp/branch-v3meta.yml`): + ```yaml + permissions: + id-token: write # enables AWS-GitHub OIDC for the best-effort v4 ingest step + contents: read + ``` +2. The `Ingest commit metadata to v3 server` step ends at `--repo-url ...` (~line 34, the last line of the file). Append the v4 block from `/tmp/branch-v3meta.yml` (the `Ingest commit metadata to v4 Postgres (best-effort)` variant -- note it writes an `empty.jsonl` and upserts the commit row only) after the v3 step. + +- [ ] **Step 5: Lint all three workflows** + +```bash +uvx yamllint --strict -c .yamllint.yaml .github/workflows/bench.yml .github/workflows/sql-benchmarks.yml .github/workflows/v3-commit-metadata.yml +``` + +Expected: exit 0. Fix any violations and re-run. + +- [ ] **Step 6: Sanity-check the v4 gating is intact on every new step** + +```bash +grep -c "GH_BENCH_INGEST_ROLE_ARN != ''" .github/workflows/bench.yml .github/workflows/sql-benchmarks.yml .github/workflows/v3-commit-metadata.yml +grep -c "continue-on-error: true" .github/workflows/bench.yml .github/workflows/sql-benchmarks.yml .github/workflows/v3-commit-metadata.yml +``` + +Expected: each file shows 3 gated steps (3 `GH_BENCH_INGEST_ROLE_ARN != ''` and 3 `continue-on-error: true`). Confirm the v3 `--server` steps were NOT modified (`git diff` shows only additions around them). + +- [ ] **Step 7: Commit** + +```bash +git add .github/workflows/bench.yml .github/workflows/sql-benchmarks.yml .github/workflows/v3-commit-metadata.yml +git commit -F - <<'EOF' +ci: add best-effort v4 Postgres dual-write step to the emitter workflows + +Insert the dormant, env-gated, continue-on-error v4 ingest step (OIDC +assume-role -> uv run post-ingest.py --postgres -> revalidate) after the v3 +--server step in bench.yml, sql-benchmarks.yml, and v3-commit-metadata.yml +(the last also gains id-token: write). The block no-ops until +GH_BENCH_INGEST_ROLE_ARN is set and can never fail the job; the v3 path is +untouched. + +Signed-off-by: Connor Tsui +EOF +``` + +--- + +## Self-Review + +- **Spec coverage:** the contract + revalidate tests are wired into the required `python-test` job (Task 1 Step 1); the testcontainer test into a docker-gated job (Task 1 Step 2); the v4 best-effort step is added to all three emitter workflows with `id-token` on v3-commit-metadata (Task 2). No pyproject/uv.lock changes. Only the 4 workflow files touched. +- **Placeholder scan:** none -- exact anchors, exact blocks (from the branch + the embedded ci.yml additions), exact commands. The one intentional `` is an explicit instruction to match the repo's pinned checkout SHA. +- **Consistency:** the v4 step gate (`GH_BENCH_INGEST_ROLE_ARN != ''` + `continue-on-error`) is identical across all three workflows; the setup-uv SHA matches between Task 1's scripts-test job and Task 2's v4 blocks; the test commands in Task 1 match the locally-verified 1.2 commands. From 0e2848371df723acd633e759c3d12a2486f53e38 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:31:10 -0400 Subject: [PATCH 22/33] ci: run the v4 emitter scripts/ tests Add the measurement_id contract test + the revalidate test to the python-test job (required, no docker), and a lightweight docker-gated scripts-test job for the testcontainer Postgres writer suite. Deps are supplied per-invocation via uv run --with (no pyproject/uv.lock changes). Signed-off-by: Connor Tsui --- .github/workflows/ci.yml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bf9c0a785b6..0422794e4e0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -113,6 +113,29 @@ jobs: uv run --all-packages make html working-directory: docs/ + - name: "Pytest - scripts (measurement_id + revalidate)" + run: >- + uv run --no-project --with pytest --with xxhash pytest + scripts/test_measurement_id.py scripts/test_post_ingest_revalidate.py + + scripts-test: + name: "Python (scripts testcontainers)" + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6 + - uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + # The testcontainer suite MUST run in CI, not silently skip: a green job with + # skipped tests would let a writer regression merge undetected. `docker info` + # fails the job up front if the daemon is unavailable, and the CI env var makes + # the testcontainer fixtures fail (not skip) per their _require_docker_for_testcontainers. + - name: "Verify Docker is available for testcontainers" + run: docker info + - name: "Pytest - scripts testcontainers" + run: >- + uv run --no-project --with pytest --with "psycopg[binary]" --with boto3 + --with xxhash --with testcontainers pytest scripts/test_post_ingest_postgres.py + rust-docs: name: "Rust (docs)" timeout-minutes: 30 From 0b54e55b2689b1857bf05d1976fb91853d9e9f47 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:35:43 -0400 Subject: [PATCH 23/33] ci: add best-effort v4 Postgres dual-write step to the emitter workflows Insert the dormant, env-gated, continue-on-error v4 ingest step (OIDC assume-role -> uv run post-ingest.py --postgres -> revalidate) after the v3 --server step in bench.yml, sql-benchmarks.yml, and v3-commit-metadata.yml (the last also gains id-token: write). The block no-ops until GH_BENCH_INGEST_ROLE_ARN is set and can never fail the job; the v3 path is untouched. Signed-off-by: Connor Tsui --- .github/workflows/bench.yml | 40 ++++++++++++++++++++++++ .github/workflows/sql-benchmarks.yml | 39 +++++++++++++++++++++++ .github/workflows/v3-commit-metadata.yml | 39 +++++++++++++++++++++++ 3 files changed, 118 insertions(+) diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index 146f0946dc0..5e3a71e9f86 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -128,6 +128,46 @@ jobs: --benchmark-id "${{ matrix.benchmark.id }}" \ --repo-url "${{ github.server_url }}/${{ github.repository }}" + # v4 (Postgres) dual-write -- BEST-EFFORT during the migration soak. The + # proven v3 step above is hard-required; a v4 failure must NOT fail the job + # (v4 is promoted to required at cutover, PR-5.1). Gated on the ingest-role + # ARN var (the assume-role input that MUST exist for OIDC to succeed) so it + # no-ops until v4 infra is wired, and every step is + # `continue-on-error` so an OIDC / uv / connect hiccup never breaks the v3 + # pipeline. post-ingest.py mints the RDS IAM token internally (boto3) from + # the assumed GitHubBenchmarkIngestRole; sslmode=verify-full validates the cert. + - name: Configure AWS credentials for v4 ingest (OIDC) + if: vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + uses: aws-actions/configure-aws-credentials@99214aa6889fcddfa57764031d71add364327e59 # v6 + with: + role-to-assume: ${{ vars.GH_BENCH_INGEST_ROLE_ARN }} + aws-region: ${{ vars.RDS_BENCH_REGION }} + - name: Install uv for v4 ingest + if: vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + - name: Ingest results to v4 Postgres (best-effort) + if: vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + shell: bash + env: + RDS_BENCH_INSTANCE_ENDPOINT: ${{ vars.RDS_BENCH_INSTANCE_ENDPOINT }} + RDS_BENCH_DB_NAME: ${{ vars.RDS_BENCH_DB_NAME }} + AWS_REGION: ${{ vars.RDS_BENCH_REGION }} + BENCH_SITE_BASE_URL: ${{ vars.BENCH_SITE_BASE_URL }} + BENCH_REVALIDATE_TOKEN: ${{ secrets.BENCH_REVALIDATE_TOKEN }} + run: | + set -Eeuo pipefail + curl -fsSL https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem \ + -o "${RUNNER_TEMP}/rds-global-bundle.pem" + DSN="postgresql://bench_ingest@${RDS_BENCH_INSTANCE_ENDPOINT}:5432/${RDS_BENCH_DB_NAME}?sslmode=verify-full&sslrootcert=${RUNNER_TEMP}/rds-global-bundle.pem" + uv run --no-project --with 'psycopg[binary]' --with boto3 --with xxhash \ + scripts/post-ingest.py results.v3.jsonl \ + --postgres "${DSN}" \ + --commit-sha "${{ github.sha }}" \ + --region "${AWS_REGION}" + - name: Alert incident.io if: failure() uses: ./.github/actions/alert-incident-io diff --git a/.github/workflows/sql-benchmarks.yml b/.github/workflows/sql-benchmarks.yml index 3686934af39..665612773b7 100644 --- a/.github/workflows/sql-benchmarks.yml +++ b/.github/workflows/sql-benchmarks.yml @@ -682,6 +682,45 @@ jobs: --benchmark-id "${{ matrix.id }}" \ --repo-url "${{ github.server_url }}/${{ github.repository }}" + # v4 (Postgres) dual-write -- BEST-EFFORT during the migration soak (see bench.yml + # for the full rationale). The v3 step above is hard-required; a v4 failure must NOT + # fail the job (v4 is promoted to required at cutover, PR-5.1). Gated on + # inputs.mode == 'develop' (matching v3) + the ingest-role ARN var (the assume-role + # input that MUST exist for OIDC to succeed; it no-ops until v4 infra is wired), and + # every step is continue-on-error. post-ingest.py mints the RDS IAM token (boto3) from + # the assumed GitHubBenchmarkIngestRole; sslmode=verify-full validates the cert. + - name: Configure AWS credentials for v4 ingest (OIDC) + if: inputs.mode == 'develop' && vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + uses: aws-actions/configure-aws-credentials@99214aa6889fcddfa57764031d71add364327e59 # v6 + with: + role-to-assume: ${{ vars.GH_BENCH_INGEST_ROLE_ARN }} + aws-region: ${{ vars.RDS_BENCH_REGION }} + - name: Install uv for v4 ingest + if: inputs.mode == 'develop' && vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + - name: Ingest results to v4 Postgres (best-effort) + if: inputs.mode == 'develop' && vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + shell: bash + env: + RDS_BENCH_INSTANCE_ENDPOINT: ${{ vars.RDS_BENCH_INSTANCE_ENDPOINT }} + RDS_BENCH_DB_NAME: ${{ vars.RDS_BENCH_DB_NAME }} + AWS_REGION: ${{ vars.RDS_BENCH_REGION }} + BENCH_SITE_BASE_URL: ${{ vars.BENCH_SITE_BASE_URL }} + BENCH_REVALIDATE_TOKEN: ${{ secrets.BENCH_REVALIDATE_TOKEN }} + run: | + set -Eeuo pipefail + curl -fsSL https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem \ + -o "${RUNNER_TEMP}/rds-global-bundle.pem" + DSN="postgresql://bench_ingest@${RDS_BENCH_INSTANCE_ENDPOINT}:5432/${RDS_BENCH_DB_NAME}?sslmode=verify-full&sslrootcert=${RUNNER_TEMP}/rds-global-bundle.pem" + uv run --no-project --with 'psycopg[binary]' --with boto3 --with xxhash \ + scripts/post-ingest.py results.v3.jsonl \ + --postgres "${DSN}" \ + --commit-sha "${{ github.sha }}" \ + --region "${AWS_REGION}" + - name: Alert incident.io if: failure() && inputs.mode == 'develop' uses: ./.github/actions/alert-incident-io diff --git a/.github/workflows/v3-commit-metadata.yml b/.github/workflows/v3-commit-metadata.yml index d95da69207b..a71b9fafcf6 100644 --- a/.github/workflows/v3-commit-metadata.yml +++ b/.github/workflows/v3-commit-metadata.yml @@ -9,6 +9,7 @@ on: workflow_dispatch: { } permissions: + id-token: write # enables AWS-GitHub OIDC for the best-effort v4 ingest step contents: read jobs: @@ -32,3 +33,41 @@ jobs: --commit-sha "${{ github.sha }}" \ --benchmark-id "commit-metadata" \ --repo-url "${{ github.server_url }}/${{ github.repository }}" + + # v4 (Postgres) dual-write -- BEST-EFFORT (see bench.yml rationale). Empty records: + # post-ingest.py --postgres upserts the commit row only. v3 above stays required; + # a v4 failure never fails the job (promoted to required at cutover, PR-5.1). + # Gated on the ingest-role ARN var (the assume-role input that MUST exist) so + # it no-ops until v4 infra is wired. + - name: Configure AWS credentials for v4 ingest (OIDC) + if: vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + uses: aws-actions/configure-aws-credentials@99214aa6889fcddfa57764031d71add364327e59 # v6 + with: + role-to-assume: ${{ vars.GH_BENCH_INGEST_ROLE_ARN }} + aws-region: ${{ vars.RDS_BENCH_REGION }} + - name: Install uv for v4 ingest + if: vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + - name: Ingest commit metadata to v4 Postgres (best-effort) + if: vars.GH_BENCH_INGEST_ROLE_ARN != '' + continue-on-error: true + shell: bash + env: + RDS_BENCH_INSTANCE_ENDPOINT: ${{ vars.RDS_BENCH_INSTANCE_ENDPOINT }} + RDS_BENCH_DB_NAME: ${{ vars.RDS_BENCH_DB_NAME }} + AWS_REGION: ${{ vars.RDS_BENCH_REGION }} + BENCH_SITE_BASE_URL: ${{ vars.BENCH_SITE_BASE_URL }} + BENCH_REVALIDATE_TOKEN: ${{ secrets.BENCH_REVALIDATE_TOKEN }} + run: | + set -Eeuo pipefail + echo -n > empty.jsonl + curl -fsSL https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem \ + -o "${RUNNER_TEMP}/rds-global-bundle.pem" + DSN="postgresql://bench_ingest@${RDS_BENCH_INSTANCE_ENDPOINT}:5432/${RDS_BENCH_DB_NAME}?sslmode=verify-full&sslrootcert=${RUNNER_TEMP}/rds-global-bundle.pem" + uv run --no-project --with 'psycopg[binary]' --with boto3 --with xxhash \ + scripts/post-ingest.py empty.jsonl \ + --postgres "${DSN}" \ + --commit-sha "${{ github.sha }}" \ + --region "${AWS_REGION}" From 6dba6e484320f977400e6e30ddab7bf06e3d718a Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 12:39:22 -0400 Subject: [PATCH 24/33] =?UTF-8?q?plan:=20set=20status=20reviewing=20?= =?UTF-8?q?=E2=80=94=20sub-phase=201.3=20gauntlet=20checkpoint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 9fbce346f34..9a7b96f8f59 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -107,7 +107,7 @@ v2/v3 paths. phase: "1: D — port v4 emitter (CODE)" sub_phase: "1.3 CI + workflow wiring" task: null -status: implementing +status: reviewing last_gate: null phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` From 38c52eb8fc5ba4c968b5f27cfbfda2f89ec4f952 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:16:11 -0400 Subject: [PATCH 25/33] plan: sub-phase 1.3 ledger entry + deferred carry-forward Record sub-phase 1.3 (CI + workflow wiring) as gauntlet pr-3 accepted (1 cycle; fresh + correctness + maint, zero must-fix). Park should-fixes: 4 cheap in-scope ones (configure-aws-credentials SHA alignment, xxhash importorskip, AWS_REGION message, revalidate method assertion) flagged to apply as a phase-D finalization polish before the phase-end gauntlet; the rest are no-action nits / pre-existing patterns. Note the scripts-test required-check as a branch-protection ops item. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 31 +++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 9a7b96f8f59..ec1d27b6d22 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -246,6 +246,26 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles implies an automated drift guard that does not exist. Deferral rationale: docs-only; the fixture CODE is column-by-column correct (opus-verified). Apply as a doc cleanup during phase-D finalization (before the phase-end gauntlet). +- **Sub-phase 1.3 — APPLY in phase-D finalization** (cheap, in-scope, clearly-correct should-fixes + to land as a polish commit BEFORE the phase-end gauntlet, so the shipped artifact is clean): + (a) `configure-aws-credentials` SHA mismatch — the 3 new v4 blocks pin `@99214aa6889… # v6` + while the repo's existing v3/S3 steps pin `@e7f100cf4c008… # v6`; align the v4 blocks to the + repo's existing `e7f100cf…` so each file is internally consistent (flagged by fresh + maint). + (b) add `pytest.importorskip("xxhash")` to `scripts/test_measurement_id.py` (the 1.1-deferred + local-dev-UX fix; CI already supplies xxhash via `--with`). (c) `scripts/post-ingest.py:~1000` + error message says "set AWS_REGION" but boto3 reads `AWS_DEFAULT_REGION` — fix the message. + (d) `scripts/test_post_ingest_revalidate.py` — assert the revalidate request method is POST + (the 1.2-deferred nit). +- **Sub-phase 1.3**, **deferred / no-action** (nits + pre-existing): contract+revalidate step + lives in the required `python-test` job (deliberate — required-check placement beats the + build-coupling concern fresh raised); `sql-benchmarks.yml` has no `id-token: write` block + (pre-existing pattern — its existing Setup-AWS-CLI step relies on the same caller-inherited + permission; all callers grant it); redundant "Install uv" step in sql-benchmarks (harmless, + idempotent, continue-on-error); v4 steps omit `--repo-url` (post-ingest.py's default is correct + for the canonical repo); `--with testcontainers` unpinned; `empty.jsonl` created in both the v3 + and v4 steps of v3-commit-metadata (defensible — the v4 step may run when the v3 step skipped). +- **Ops (not code)**: making the new `scripts-test` job a REQUIRED check is a GitHub + branch-protection setting (admin action), not a code change — track alongside the ops phases. - **Sub-phase 1.2**, multiple files, **nits** (~10): test rollback-assertion consistency + revalidate HTTP-method assertion (fresh); non-string `commit_sha` yields a misleading mismatch error (correctness); plan-cycle/PR labels in comments, `_load_module` 4x duplication, handler @@ -277,3 +297,14 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles postgres:16-alpine. - **Gauntlet:** pr-3 / accepted (cycles: 1) — fresh + correctness (opus) + maint, zero must-fix. - **Deferred:** 4 should-fix-class + ~10 nits (see Carry-forward > Deferred work) + +#### Sub-phase 1.3: CI + workflow wiring + +- **Shipped:** wired the contract + revalidate tests into the required `python-test` CI job and + the testcontainer suite into a new docker-gated `scripts-test` job (all via `uv run --with`, no + pyproject/uv.lock changes); added the dormant best-effort v4 ingest step block to bench.yml, + sql-benchmarks.yml, v3-commit-metadata.yml (+ `id-token: write`), gated on + `vars.GH_BENCH_INGEST_ROLE_ARN != ''` + `continue-on-error`. +- **Gauntlet:** pr-3 / accepted (cycles: 1) — fresh + correctness + maint, zero must-fix. +- **Deferred:** 4 apply-in-finalization should-fixes + several no-action nits/pre-existing + (see Carry-forward > Deferred work). From 4da50e18633b415335c64c5c4507bc6c8bade45f Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:16:43 -0400 Subject: [PATCH 26/33] =?UTF-8?q?plan:=20all=20Phase=201=20sub-phases=20co?= =?UTF-8?q?mplete=20=E2=80=94=20enter=20Phase=203=20(phase=20boundary)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sub-phase 1.3 done + gauntlet-accepted; clear sub_phase (status stays reviewing) to route into the Phase 3 phase-end review. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index ec1d27b6d22..4dc08e005b2 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -105,7 +105,7 @@ v2/v3 paths. ```yaml phase: "1: D — port v4 emitter (CODE)" -sub_phase: "1.3 CI + workflow wiring" +sub_phase: null task: null status: reviewing last_gate: null From 6fdd727f0a78e0bdf6d9cdd70b7f0b40fe1b1311 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:19:40 -0400 Subject: [PATCH 27/33] fix: phase-D finalization polish (gauntlet should-fixes) Align the new v4 configure-aws-credentials steps to the repo's existing v6 SHA pin (one SHA per file); add pytest.importorskip("xxhash") so the measurement_id test skips gracefully without xxhash locally; correct the RDS region error message to AWS_DEFAULT_REGION (what boto3 reads); assert the revalidate request method is POST. Signed-off-by: Connor Tsui --- .github/workflows/bench.yml | 2 +- .github/workflows/sql-benchmarks.yml | 2 +- .github/workflows/v3-commit-metadata.yml | 2 +- scripts/post-ingest.py | 4 +++- scripts/test_measurement_id.py | 1 + scripts/test_post_ingest_revalidate.py | 8 +++++--- 6 files changed, 12 insertions(+), 7 deletions(-) diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index 5e3a71e9f86..cc26c6975ad 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -139,7 +139,7 @@ jobs: - name: Configure AWS credentials for v4 ingest (OIDC) if: vars.GH_BENCH_INGEST_ROLE_ARN != '' continue-on-error: true - uses: aws-actions/configure-aws-credentials@99214aa6889fcddfa57764031d71add364327e59 # v6 + uses: aws-actions/configure-aws-credentials@e7f100cf4c008499ea8adda475de1042d6975c7b # v6 with: role-to-assume: ${{ vars.GH_BENCH_INGEST_ROLE_ARN }} aws-region: ${{ vars.RDS_BENCH_REGION }} diff --git a/.github/workflows/sql-benchmarks.yml b/.github/workflows/sql-benchmarks.yml index 665612773b7..ffefa28b802 100644 --- a/.github/workflows/sql-benchmarks.yml +++ b/.github/workflows/sql-benchmarks.yml @@ -692,7 +692,7 @@ jobs: - name: Configure AWS credentials for v4 ingest (OIDC) if: inputs.mode == 'develop' && vars.GH_BENCH_INGEST_ROLE_ARN != '' continue-on-error: true - uses: aws-actions/configure-aws-credentials@99214aa6889fcddfa57764031d71add364327e59 # v6 + uses: aws-actions/configure-aws-credentials@e7f100cf4c008499ea8adda475de1042d6975c7b # v6 with: role-to-assume: ${{ vars.GH_BENCH_INGEST_ROLE_ARN }} aws-region: ${{ vars.RDS_BENCH_REGION }} diff --git a/.github/workflows/v3-commit-metadata.yml b/.github/workflows/v3-commit-metadata.yml index a71b9fafcf6..0e411730db8 100644 --- a/.github/workflows/v3-commit-metadata.yml +++ b/.github/workflows/v3-commit-metadata.yml @@ -42,7 +42,7 @@ jobs: - name: Configure AWS credentials for v4 ingest (OIDC) if: vars.GH_BENCH_INGEST_ROLE_ARN != '' continue-on-error: true - uses: aws-actions/configure-aws-credentials@99214aa6889fcddfa57764031d71add364327e59 # v6 + uses: aws-actions/configure-aws-credentials@e7f100cf4c008499ea8adda475de1042d6975c7b # v6 with: role-to-assume: ${{ vars.GH_BENCH_INGEST_ROLE_ARN }} aws-region: ${{ vars.RDS_BENCH_REGION }} diff --git a/scripts/post-ingest.py b/scripts/post-ingest.py index 38c352dea3d..e349048f04b 100755 --- a/scripts/post-ingest.py +++ b/scripts/post-ingest.py @@ -997,7 +997,9 @@ def _rds_iam_token(*, host: str, port: int, user: str, region: str | None) -> st session = boto3.session.Session() resolved = region or session.region_name or _region_from_host(host) if not resolved: - raise SystemExit("could not determine the AWS region for the RDS IAM token; pass --region or set AWS_REGION.") + raise SystemExit( + "could not determine the AWS region for the RDS IAM token; pass --region or set AWS_DEFAULT_REGION." + ) client = session.client("rds", region_name=resolved) return client.generate_db_auth_token(DBHostname=host, Port=port, DBUsername=user, Region=resolved) diff --git a/scripts/test_measurement_id.py b/scripts/test_measurement_id.py index 88a40202b86..d4c1bd2b471 100644 --- a/scripts/test_measurement_id.py +++ b/scripts/test_measurement_id.py @@ -38,6 +38,7 @@ def _load_port(): return module +pytest.importorskip("xxhash") port = _load_port() diff --git a/scripts/test_post_ingest_revalidate.py b/scripts/test_post_ingest_revalidate.py index 341de20efdf..21a5216aac1 100644 --- a/scripts/test_post_ingest_revalidate.py +++ b/scripts/test_post_ingest_revalidate.py @@ -44,10 +44,10 @@ def __exit__(self, *exc): def test_refresh_posts_revalidate_with_bearer(monkeypatch): - calls: list[tuple[str, dict[str, str], bytes | None]] = [] + calls: list[tuple[str, str, dict[str, str], bytes | None]] = [] def fake_urlopen(req, timeout=None): - calls.append((req.full_url, dict(req.headers), req.data)) + calls.append((req.full_url, req.get_method(), dict(req.headers), req.data)) return _FakeResponse(b'{"groups": []}') monkeypatch.setattr(post_ingest.urllib.request, "urlopen", fake_urlopen) @@ -55,8 +55,10 @@ def fake_urlopen(req, timeout=None): revalidate = [c for c in calls if c[0].endswith("/api/revalidate")] assert revalidate, "expected a POST to /api/revalidate" + # The revalidate request must use the POST method. + assert revalidate[0][1] == "POST", "revalidate must be a POST request" # urllib title-cases header keys, so the bearer lives under "Authorization". - assert revalidate[0][1].get("Authorization") == "Bearer tok" + assert revalidate[0][2].get("Authorization") == "Bearer tok" # Revalidate must be the first request issued, before any warm GETs. assert calls[0][0].endswith("/api/revalidate"), "revalidate must precede warm GETs" From c35badc7d9d34dac704e05e7e74bee4447df876f Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:32:07 -0400 Subject: [PATCH 28/33] =?UTF-8?q?plan:=20Phase=201=20gate=20=E2=80=94=20ph?= =?UTF-8?q?ase-4=20gauntlet=20accepted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the Phase 1 (D) gate: exit criteria all pass; phase-end gauntlet phase-4 accepted (spec + correctness + maint + arch, zero must-fix; 172 tests re-run green). Fix the stale Accepted-tradeoffs PEP-723 wording (deps are via uv run --with, not PEP-723). Park the phase-4 refactor/doc should-fixes in Deferred work. status stays reviewing until the phase PR is opened. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 37 +++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 4dc08e005b2..bf2a07c496d 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -204,8 +204,9 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles #### Accepted tradeoffs - **v4 emitter third-party deps**: the v4 `--postgres` path depends on third-party packages - (psycopg + IAM-token minting) declared in `post-ingest.py`'s PEP-723 block and lazily imported - inside the postgres branch. Accepted: the v3 `--server` path stays stdlib-only and unaffected. + (`psycopg[binary]`, `boto3`, `xxhash`) supplied at the call site via `uv run --no-project --with` + (NOT declared in `post-ingest.py`'s PEP-723 block, which stays `dependencies = []`), and lazily + imported inside the postgres branch. Accepted: the v3 `--server` path stays stdlib-only. - **v4 step best-effort by design**: every v4 workflow step is `continue-on-error: true` and env-gated. Accepted: a v4 failure intentionally does not fail the workflow (additive write). @@ -266,6 +267,26 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles and v4 steps of v3-commit-metadata (defensible — the v4 step may run when the v3 step skipped). - **Ops (not code)**: making the new `scripts-test` job a REQUIRED check is a GitHub branch-protection setting (admin action), not a code change — track alongside the ops phases. +- **Phase 1 gate (phase-4) review** — new should-fixes (all non-blocking; refactor/hardening + opportunities for a follow-up): (a) [arch] the 3-step v4 ingest block is copy-pasted across + bench/sql-benchmarks/v3-commit-metadata (the SHA mismatch was a symptom) — consider extracting a + `.github/actions/v4-ingest` composite action to collapse the DSN/CA-bundle/SHA to one site; + (b) [arch/correctness] the record-kind->measurement_id dispatch + record-schema maps are restated + across post-ingest.py, cross_check, and the tests — consolidate when the next fact table lands + (the test maps are deliberately independent for verification value); (c) [correctness] commit-dict + fields are accessed without the loud record-indexed validation convention (a malformed timestamp + would raise a raw Postgres error; relates to the 1.2 cross_check KeyError item) — harden with the + same convention when cross_check is next touched. Deferral rationale: the production path is safe + (build_commit always supplies all fields; the workflow blocks are now SHA-consistent); these are + maintainability refactors, not correctness gaps. +- **Phase 1 gate — no-action**: [correctness] `query_idx`/`iterations` validated as i32 (matching + the v3 server's storage + the measurement_id hash) vs the producer's u32 wire type — not + practically reachable. Doc nits (fold into a future doc pass): `measurement_id_golden.json` `note` + + `_measurement_id.py` docstring still say `benchmarks-website/.../db.rs` without the + `vortex-data/` org prefix and attribute the golden generator to `db.rs` vs `measurement_id_golden.rs`; + the fixture header + test docstring omit the 006 engine/format indexes (present in the fixture); + `post-ingest.py` comment references `migrations/004`; the v4 blocks lack a machine-checkable removal + trigger comment. - **Sub-phase 1.2**, multiple files, **nits** (~10): test rollback-assertion consistency + revalidate HTTP-method assertion (fresh); non-string `commit_sha` yields a misleading mismatch error (correctness); plan-cycle/PR labels in comments, `_load_module` 4x duplication, handler @@ -308,3 +329,15 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles - **Gauntlet:** pr-3 / accepted (cycles: 1) — fresh + correctness + maint, zero must-fix. - **Deferred:** 4 apply-in-finalization should-fixes + several no-action nits/pre-existing (see Carry-forward > Deferred work). + +#### Phase 1 gate + +- **Phase-D finalization polish:** applied the 4 in-scope should-fixes (configure-aws-credentials + SHA aligned to `e7f100cf`, `xxhash` importorskip, `AWS_DEFAULT_REGION` message, revalidate POST + assertion) in commit `6fdd727f0`, reviewed by the phase-end gauntlet. +- **Exit criteria:** all PASS — measurement_id contract (65), yamllint --strict (4 workflows), + `post-ingest.py --help` stdlib-only, testcontainer writer suite (100) under docker. +- **Gauntlet:** phase-4 / accepted (cycles: 1) — spec + correctness (opus) + maint + arch, zero + must-fix. Correctness reviewer independently re-ran 172 tests (green); arch confirmed the v4 + path can never break the live v3/v2 path (dormant-but-ready). New should-fixes are refactor / + doc items deferred to follow-ups (see Carry-forward > Deferred work). From 9c4607bfdf7f468f24cc12de05ceae40ee632e41 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:37:43 -0400 Subject: [PATCH 29/33] =?UTF-8?q?plan:=20session=20handoff=20=E2=80=94=20P?= =?UTF-8?q?hase=201=20done,=20open=20decision=20=3D=20phase-D=20PR=20form?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record a SESSION HANDOFF brief at the top of the spine for a clean fresh-session resume: Phase 1 (D) is complete + phase-4 gauntlet accepted (gate in the ledger); the open decision is the phase-D PR form (code-only [recommended] / include scaffolding / hold); do not re-run the phase-end gauntlet on resume; demo-safety and 1Password constraints still in force. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 52 +++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index bf2a07c496d..fac01945fc4 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -4,6 +4,58 @@ **Design spec:** `.big-plans/ct__bench-v4-emitters-design.md` (written by brainstorming in Step 1.2) **Work shape:** feature-integration +--- + +## SESSION HANDOFF (2026-06-19) — READ FIRST ON RESUME + +**Where we are:** Phase 1 (D, the CODE phase) is COMPLETE and gauntlet-accepted end-to-end. All +3 sub-phases shipped + checkpoint-accepted (1.1 measurement_id contract pr-2; 1.2 Postgres writer +pr-3; 1.3 CI + workflow wiring pr-3); the phase-D finalization polish landed (`6fdd727f0`); the +phase-end gauntlet (phase-4: spec + correctness + maint + arch) ACCEPTED with zero must-fix (172 +tests re-run green); the Phase 1 gate is recorded in the Verdict Ledger. HEAD = `c35badc7d`, +pushed to `origin/ct/bench-v4-emitters`. Current Position reads `status: reviewing` / +`sub_phase: null` ONLY because the phase PR has not been opened yet (that is the next step). + +**DO NOT re-run the phase-end gauntlet on resume** — it already passed and is recorded in the +ledger (`#### Phase 1 gate`, phase-4 / accepted). The stock `reviewing` + `sub_phase: null` route +would re-run it; skip that and go straight to the OPEN DECISION below. + +**OPEN DECISION (RE-ASK THE USER — the AUQ was interrupted):** how to open the phase-D PR (Step 3.3): +- **(A, recommended) CODE-ONLY PR** — the 8 code commits touch only `scripts/` + `.github/` + (range `ec31812ce..6fdd727f0`, excluding the `plan:` commits which touch only `.big-plans/`). + Cherry-pick them onto a fresh branch off `develop`; open a draft PR from there. Keep the spine + branch-local on `ct/bench-v4-emitters`. Requires a Step 3.5 deviation: after the code PR merges, + do NOT `git reset --hard origin/develop` on the orchestration branch (it would wipe the spine) — + reset then restore/re-commit the `.big-plans/` spine. +- **(B) INCLUDE SCAFFOLDING (big-plans default)** — open the draft PR from `ct/bench-v4-emitters` + as-is (28 commits, `.big-plans/` in the diff); spine rides to develop transiently, removed by a + dedicated wrap-up PR. +- **(C) HOLD** — don't open the PR yet (e.g. to avoid PR CI / team noise during the demo). + +**After the PR-form decision:** open the draft PR (body per `spiral:pr-and-issue-voice`: what +shipped per sub-phase, gauntlet verdict, exit criteria, deferred items), commit +`plan: phase PR opened — #`, set `status: awaiting-human-gate`, fire the Step 3.4 human gate +(proceed / re-plan / amend / abort). On "proceed" → Step 3.5 merges phase D, then phases A -> C -> B. + +**HARD CONSTRAINTS still in force (design spec §4.0):** +- **DEMO SAFETY:** NO prod RDS writes until phase B. Phases C and B are GATED — post-demo + + explicit user go-ahead. Only phase D (this code, done) was demo-safe. Confirm the demo is over + before any prod-touching op (A is data-safe IAM-only but still an external mutation; C/B touch + prod/site). +- **1PASSWORD must be UNLOCKED** for commit/push (SSH signing). It auto-locked mid-session and + blocked a commit; if commits fail with `1Password: failed to fill whole buffer`, ask the user + to unlock the desktop app. +- **Remaining phases after D merges:** A (create `GitHubBenchmarkIngestRole`, ops, data-safe), + C (align `BENCH_REVALIDATE_TOKEN` on Vercel prod + monorepo secret + redeploy v4, ops, GATED, + post-demo), B (set `GH_BENCH_INGEST_ROLE_ARN` + repoint `BENCH_SITE_BASE_URL`/`BENCHMARKS_WEB_PROD_URL`, + live cutover + acceptance §6, ops, GATED, post-demo). All ops phases: direct CLI, no gauntlet/PR, + pre-action confirm. See Orchestration notes + design spec §4-§6. + +**Resume:** re-invoke `/spiral:big-plans` on branch `ct/bench-v4-emitters`; Phase 0 reads this +handoff + the Current Position below. + +--- + ## Goal Make the Vortex monorepo CI emitters write benchmark results LIVE to the v4 RDS Postgres From 3301626b57efea6d2f2127ddca16580ca7d7b722 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:52:32 -0400 Subject: [PATCH 30/33] =?UTF-8?q?plan:=20phase=20PR=20opened=20=E2=80=94?= =?UTF-8?q?=20#8512?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the phase-D PR (#8512, draft -> develop) in the Verdict Ledger and refresh the SESSION HANDOFF brief: the open PR-form decision is resolved (user chose include-scaffolding, so .big-plans/ rides in the diff), and the next step is the Step 3.4 human gate. status stays reviewing here; the awaiting-human-gate transition is the next commit. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 41 +++++++++++------------------ 1 file changed, 15 insertions(+), 26 deletions(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index fac01945fc4..433af040dbb 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -8,34 +8,21 @@ ## SESSION HANDOFF (2026-06-19) — READ FIRST ON RESUME -**Where we are:** Phase 1 (D, the CODE phase) is COMPLETE and gauntlet-accepted end-to-end. All -3 sub-phases shipped + checkpoint-accepted (1.1 measurement_id contract pr-2; 1.2 Postgres writer -pr-3; 1.3 CI + workflow wiring pr-3); the phase-D finalization polish landed (`6fdd727f0`); the -phase-end gauntlet (phase-4: spec + correctness + maint + arch) ACCEPTED with zero must-fix (172 -tests re-run green); the Phase 1 gate is recorded in the Verdict Ledger. HEAD = `c35badc7d`, -pushed to `origin/ct/bench-v4-emitters`. Current Position reads `status: reviewing` / -`sub_phase: null` ONLY because the phase PR has not been opened yet (that is the next step). +**Where we are:** Phase 1 (D, the CODE phase) is COMPLETE and gauntlet-accepted end-to-end (all +3 sub-phases checkpoint-accepted; the phase-end phase-4 gauntlet ACCEPTED with zero must-fix, 172 +tests re-run green; the Phase 1 gate is recorded in the Verdict Ledger). The phase-D PR is OPEN: +**#8512** (draft -> develop, INCLUDE-SCAFFOLDING form chosen by the user, so `.big-plans/` rides +in the diff). `status: awaiting-human-gate`. The user indicated they will likely merge after CI +goes green. **DO NOT re-run the phase-end gauntlet on resume** — it already passed and is recorded in the -ledger (`#### Phase 1 gate`, phase-4 / accepted). The stock `reviewing` + `sub_phase: null` route -would re-run it; skip that and go straight to the OPEN DECISION below. - -**OPEN DECISION (RE-ASK THE USER — the AUQ was interrupted):** how to open the phase-D PR (Step 3.3): -- **(A, recommended) CODE-ONLY PR** — the 8 code commits touch only `scripts/` + `.github/` - (range `ec31812ce..6fdd727f0`, excluding the `plan:` commits which touch only `.big-plans/`). - Cherry-pick them onto a fresh branch off `develop`; open a draft PR from there. Keep the spine - branch-local on `ct/bench-v4-emitters`. Requires a Step 3.5 deviation: after the code PR merges, - do NOT `git reset --hard origin/develop` on the orchestration branch (it would wipe the spine) — - reset then restore/re-commit the `.big-plans/` spine. -- **(B) INCLUDE SCAFFOLDING (big-plans default)** — open the draft PR from `ct/bench-v4-emitters` - as-is (28 commits, `.big-plans/` in the diff); spine rides to develop transiently, removed by a - dedicated wrap-up PR. -- **(C) HOLD** — don't open the PR yet (e.g. to avoid PR CI / team noise during the demo). - -**After the PR-form decision:** open the draft PR (body per `spiral:pr-and-issue-voice`: what -shipped per sub-phase, gauntlet verdict, exit criteria, deferred items), commit -`plan: phase PR opened — #`, set `status: awaiting-human-gate`, fire the Step 3.4 human gate -(proceed / re-plan / amend / abort). On "proceed" → Step 3.5 merges phase D, then phases A -> C -> B. +ledger (`#### Phase 1 gate`, phase-4 / accepted). + +**NEXT (Step 3.4 / 3.5 human gate):** on "proceed", squash-merge #8512 -> stable-branch sync -> +advance to phase A. If the user squash-merges #8512 on GitHub themselves, reconcile the spine +afterward (the squash carries the spine + gate records onto develop) and advance to phase A. +Before touching AWS in phase A, CONFIRM the demo is over (phase A is data-safe IAM-only but is +still an external mutation) per the HARD CONSTRAINTS below. **HARD CONSTRAINTS still in force (design spec §4.0):** - **DEMO SAFETY:** NO prod RDS writes until phase B. Phases C and B are GATED — post-demo + @@ -393,3 +380,5 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles must-fix. Correctness reviewer independently re-ran 172 tests (green); arch confirmed the v4 path can never break the live v3/v2 path (dormant-but-ready). New should-fixes are refactor / doc items deferred to follow-ups (see Carry-forward > Deferred work). +- **Phase PR:** #8512 (draft -> develop; include-scaffolding form, user-chosen). Opened + 2026-06-19; CI running; the plan is to squash-merge on green. From bd2b10ba87db9a8fb786026a619f25bcf5e595c4 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 13:52:48 -0400 Subject: [PATCH 31/33] =?UTF-8?q?plan:=20set=20status=20awaiting-human-gat?= =?UTF-8?q?e=20=E2=80=94=20phase=201=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three phase-boundary gate prerequisites now exist: exit criteria passed, the phase-end gauntlet accepted, and the phase PR (#8512) number is committed. Move Current Position to awaiting-human-gate so a resume re-presents the gate. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 433af040dbb..5af7d4e4ce9 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -146,7 +146,7 @@ v2/v3 paths. phase: "1: D — port v4 emitter (CODE)" sub_phase: null task: null -status: reviewing +status: awaiting-human-gate last_gate: null phase_entry_sha: a3ffeeea8ad1c9147b31a0a4ece5233143975f32 ``` From 05b6b79f6b3757d76724a5a5961984df0c602354 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 14:01:23 -0400 Subject: [PATCH 32/33] ci: green the phase-D PR checks (REUSE + typos) Opening the phase-D PR surfaced two real PR-only check failures, both fixed here without touching any reviewed code: - reuse-check: the .big-plans/ orchestration scaffolding (spine + task plans) that rides in this PR lacks SPDX headers. Add a REUSE.toml annotation licensing .big-plans/** as CC-BY-4.0, matching docs/ and .agents/skills/. The annotation is removed by the scaffolding-cleanup PR later. - typos: the spell checker mis-splits the SQL verbs UPDATEs / UPDATEd / INSERTs / INSERTed in scripts/cross_check_python_writer.py docstrings (UPDATEs -> UPDAT). Add those four identifiers to _typos.toml's python ignore list rather than rewording the reviewed docstrings. Verified locally: reuse lint compliant (2734/2734) and typos exits 0. Signed-off-by: Connor Tsui --- REUSE.toml | 7 +++++++ _typos.toml | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/REUSE.toml b/REUSE.toml index fb3079a046d..c1c0f62f8df 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -35,6 +35,13 @@ path = ".agents/skills/**" SPDX-FileCopyrightText = "Copyright the Vortex contributors" SPDX-License-Identifier = "CC-BY-4.0" +# big-plans orchestration scaffolding (the spine + per-sub-phase task plans). Transient; removed +# by a dedicated cleanup PR once the trailing ops phases complete. +[[annotations]] +path = ".big-plans/**" +SPDX-FileCopyrightText = "Copyright the Vortex contributors" +SPDX-License-Identifier = "CC-BY-4.0" + [[annotations]] path = ["**/.gitignore", ".gitmodules", ".python-version", "**/*.lock", "**/*.lockfile", "**/*.toml", "**/*.json", "CITATION.cff", ".idea/**", ".github/**", "codecov.yml", "java/gradle/wrapper/gradle-wrapper.properties", "**.duckdb*"] precedence = "override" diff --git a/_typos.toml b/_typos.toml index e2a5ff330b8..6d0b681b022 100644 --- a/_typos.toml +++ b/_typos.toml @@ -13,4 +13,10 @@ extend-exclude = ["/vortex-bench/**", "/docs/references.bib", "benchmarks/**", " [type.py] extend-ignore-identifiers-re = [ "Trafic", # Appears in some clickbench-related examples + # SQL verbs in scripts/cross_check_python_writer.py docstrings; typos mis-splits the + # all-caps-to-lowercase boundary (UPDATEs -> UPDAT, INSERTs -> INSER). + "UPDATEs", + "UPDATEd", + "INSERTs", + "INSERTed", ] From 9e601fb05c8747315e45378860ce6fd81d023243 Mon Sep 17 00:00:00 2001 From: Connor Tsui Date: Fri, 19 Jun 2026 14:01:47 -0400 Subject: [PATCH 33/33] plan: note post-PR-open CI fixes at the phase 1 gate Record the config-only reuse-check + Spell Check fixes (commit 05b6b79f6) in the Verdict Ledger, plus the non-blocking DCO mismatch and the changelog label, so the gate audit trail reflects why the final PR state differs from the gauntlet-accepted tip. Signed-off-by: Connor Tsui --- .big-plans/ct__bench-v4-emitters.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.big-plans/ct__bench-v4-emitters.md b/.big-plans/ct__bench-v4-emitters.md index 5af7d4e4ce9..6504f974a25 100644 --- a/.big-plans/ct__bench-v4-emitters.md +++ b/.big-plans/ct__bench-v4-emitters.md @@ -382,3 +382,11 @@ This spine mixes one CODE phase with three OPS phases; the orchestrator handles doc items deferred to follow-ups (see Carry-forward > Deferred work). - **Phase PR:** #8512 (draft -> develop; include-scaffolding form, user-chosen). Opened 2026-06-19; CI running; the plan is to squash-merge on green. +- **Post-PR-open CI fixes** (commit `05b6b79f6`, config-only, no reviewed-code change, so the + phase-4 gauntlet was NOT re-run): (a) `REUSE.toml` annotation licensing `.big-plans/**` + CC-BY-4.0 (the include-scaffolding files lacked SPDX headers -> reuse-check); (b) `_typos.toml` + ignore for the SQL verbs `UPDATEs`/`UPDATEd`/`INSERTs`/`INSERTed` in + `cross_check_python_writer.py` docstrings (typos mis-split -> Spell Check). Verified locally + (reuse lint compliant, typos exit 0). DCO fails on the gmail-author / spiraldb-signoff mismatch + but is non-blocking here -- develop's own latest commit carries the identical mismatch and + merged. Changelog label `changelog/ci` added to the PR.