diff --git a/docs/audit/phase3b-promotion/replacement/01_DRIFT_FINDING.md b/docs/audit/phase3b-promotion/replacement/01_DRIFT_FINDING.md new file mode 100644 index 00000000..59b9397b --- /dev/null +++ b/docs/audit/phase3b-promotion/replacement/01_DRIFT_FINDING.md @@ -0,0 +1,64 @@ +# Phase 3B — Baseline Drift Finding (why PR #248 is replaced) + +> **EVIDENCE — DOCS ONLY.** Establishes that production has schema drift not captured by +> `initDatabase()` or the migration files, so the PR #248 baseline is unfaithful. All +> diagnostics ran on **throwaway PG16** (torn down); production untouched. + +## 1. Apples-to-apples counts (same catalog queries both sides) + +The B3 operator report and PR #248 originally used **different** count methods (drizzle +introspect summary vs SQL catalog). To settle whether the gap was real or a measurement +artifact, `initDatabase()` was re-materialized on a throwaway PG16 and counted with the +**exact catalog queries the operator used**: + +| Metric (catalog query) | `initDatabase()` | Real prod (operator, PG18) | Delta | +|---|---|---|---| +| `information_schema.tables` (BASE, public) | 125 | 125 | **0** | +| `information_schema.columns` (public) | **1595** | **1649** | **+54** | +| `pg_indexes` (public) | 400 | 402 | +2 | +| non-PK indexes (`pg_indexes`, not `%_pkey`) | 275 | 277 | +2 | +| FKs (`table_constraints` FK, public) | 267 | 266 | −1 | + +The `+54` columns are **not** a measurement artifact — measured identically, `initDatabase()` +genuinely produces 54 fewer columns than production has. + +## 2. Do the hand-written migrations explain it? No. + +All **24** hand-written migration `.sql` files +(`packages/db-client/src/postgres/migrations/*.sql` + `apps/backend/migrations/*.sql`) were +applied on top of the `initDatabase()` base and columns re-counted: + +``` +initDatabase() alone ................... 1595 columns / 267 FKs ++ all 24 migration .sql files .......... 1596 columns / 267 FKs (+1 column only) +real production (Gate 1 / PG18) ........ 1649 columns / 266 FKs +``` + +The migrations add **one** column. So **~53 production columns and the FK/index deltas exist +in NO source of truth** — not the current `initDatabase()` code, not the migration files. + +## 3. Interpretation + +This is **undocumented production schema drift** — columns/indexes added directly to prod (or +by since-removed code) that the current source no longer reproduces, minus one FK the source +declares but prod lacks. It is precisely the class of drift Phase 3 exists to eliminate, and +it means: + +- ❌ A baseline generated from `initDatabase()` (PR #248) **misrepresents** production by ~54 + columns — adopting it (mark-as-applied) would record a false "this is prod" baseline and + desync future `drizzle-kit generate` diffs. +- ✅ The **only** faithful baseline is one **introspected from the real production schema** + (the Gate 1 snapshot) — which the operator already produced on PG18 in B3 (1649 columns). + +## 4. Consequence + +- **PR #248 baseline → superseded.** Do not adopt it. +- **Authoritative baseline = the PG18 introspect of the real Gate 1 snapshot** (doc 02). +- The **exact 54-column / 2-index / 1-FK delta list** should be captured (normalized diff, + doc 02 Step 3) and filed — it is a valuable inventory of the undocumented prod drift and + feeds the Phase 3C convergence work. + +## 5. Guardrails + +Diagnostics ran on throwaway PG16 only (dropped after). No production deploy, migration +execution against prod, schema application, backend wiring, or feature-flag activation. diff --git a/docs/audit/phase3b-promotion/replacement/02_REPLACEMENT_RUNBOOK.md b/docs/audit/phase3b-promotion/replacement/02_REPLACEMENT_RUNBOOK.md new file mode 100644 index 00000000..f330cbc1 --- /dev/null +++ b/docs/audit/phase3b-promotion/replacement/02_REPLACEMENT_RUNBOOK.md @@ -0,0 +1,115 @@ +# Phase 3B — Baseline Replacement Runbook (operator, PG18) + +> **OPERATOR RUNBOOK — DOCS ONLY.** Regenerates the authoritative `0000_prod_baseline` from +> the **real Gate 1 production snapshot on PostgreSQL 18**, diffs it against the superseded +> PR #248 candidate, builds the journal, and proves seeded `db:migrate` skips with zero DDL. +> Runs on throwaway PG18; **production is never touched.** + +Inputs: `postgres:18-alpine` · Gate 1 snapshot +`C:\Users\vitor\zf-phase3b\20260705T215941Z\zonforge_prod_schema.sql` · repo checkout. +Reference (real prod, from B3): **125 tables · 1649 columns · 402 indexes · 266 FKs**. + +## Step 1 — Regenerate the authoritative baseline (PG18, real snapshot) + +```powershell +docker run -d --name zf-repl -e POSTGRES_PASSWORD=repl -p 55451:5432 postgres:18-alpine +Start-Sleep 6 +docker cp "C:\Users\vitor\zf-phase3b\20260705T215941Z\zonforge_prod_schema.sql" zf-repl:/snap.sql +docker exec zf-repl psql -U postgres -v ON_ERROR_STOP=1 -f /snap.sql *> $null # load real prod schema + +# introspect into the AUTHORITATIVE baseline +@' +import type { Config } from "drizzle-kit" +export default { schema: "./packages/db-client/src/postgres/prod-baseline-v2-schema.ts", + out: "./packages/db-client/src/postgres/prod-baseline-v2", dialect: "postgresql", + dbCredentials: { url: process.env.DATABASE_URL ?? "" }, strict: true } satisfies Config +'@ | Set-Content packages/db-client/drizzle.config.replace.ts +$env:DATABASE_URL = "postgresql://postgres:repl@localhost:55451/postgres" +Remove-Item -Recurse -Force packages/db-client/src/postgres/prod-baseline-v2 -ErrorAction SilentlyContinue +npx drizzle-kit introspect --config packages/db-client/drizzle.config.replace.ts +``` +Expect the introspect summary to reflect the **real prod** (≈125 tables / 1649 cols / +402 idx / 266 FK). Rename the generated `0000_*.sql` → `0000_prod_baseline.sql` and set the +`meta/_journal.json` tag to `0000_prod_baseline`. + +## Step 2 — Fix the introspect artifact + +Correct the known `int4_ops`-on-`timestamptz` index (drizzle-kit introspect quirk) so the +DDL is valid if ever applied to a fresh DB. Grep the generated SQL for `int4_ops` and set the +correct operator class (or drop the erroneous `USING` clause) for that index. + +## Step 3 — Normalized structural diff vs PR #248 (capture the drift inventory) + +```powershell +$V2 = "packages/db-client/src/postgres/prod-baseline-v2/0000_prod_baseline.sql" +$PR248 = "docs/audit/phase3b-promotion/exec/prod-baseline/0000_prod_baseline.sql" # PR #248 branch +function Norm($f){ (Get-Content $f) -match '^\s*(CREATE TABLE|\t"|ALTER TABLE|ADD CONSTRAINT|CONSTRAINT|FOREIGN KEY|PRIMARY KEY)' | Sort-Object } +Compare-Object (Norm $PR248) (Norm $V2) | Format-Table +``` +- `=>` rows = present only in **real prod** (the ~54 drift columns / +2 indexes) — **save this + list** as the drift inventory (feeds Phase 3C). +- `<=` rows = present only in **PR #248** (e.g., the 1 FK prod lacks) — record for reconciliation. + +## Step 4 — Build + confirm the journal + +`meta/_journal.json` (from introspect) must have one `0000_prod_baseline` entry. Record its +`when` and `sha256sum 0000_prod_baseline.sql` (both feed the seed). + +## Step 5 — Prove seeded `db:migrate` skips with ZERO DDL (PG18) + +The B3 attempt was blocked because the live `meta/_journal.json` was absent. Point the +migrator at the new baseline folder: + +```powershell +# copy the v2 baseline into the path migrate.ts reads, in a THROWAWAY checkout/build +Copy-Item packages/db-client/src/postgres/prod-baseline-v2/0000_prod_baseline.sql packages/db-client/src/postgres/migrations/ +New-Item -ItemType Directory -Force packages/db-client/src/postgres/migrations/meta | Out-Null +Copy-Item packages/db-client/src/postgres/prod-baseline-v2/meta/_journal.json packages/db-client/src/postgres/migrations/meta/ +npm run build --workspace=packages/db-client # copies migrations into dist (add copy step if needed) + +$U = "postgresql://postgres:repl@localhost:55451/postgres" # PG18 with the 125 real tables +# F1 +docker exec zf-repl psql -U postgres -At -c "SELECT md5(string_agg(table_name||'.'||column_name||':'||data_type,',' ORDER BY table_name,ordinal_position)) FROM information_schema.columns WHERE table_schema='public';" +# seed the journal row (created_at = journal when), then run db:migrate -> must SKIP +docker exec zf-repl psql -U postgres -c "CREATE SCHEMA IF NOT EXISTS drizzle; CREATE TABLE IF NOT EXISTS drizzle.__drizzle_migrations(id serial primary key, hash text not null, created_at bigint); INSERT INTO drizzle.__drizzle_migrations(hash,created_at) VALUES ('', );" +$env:ZONFORGE_POSTGRES_URL = $U +npm run db:migrate --workspace=packages/db-client # expect: nothing applied (skip) -> ZERO DDL +# F2 +docker exec zf-repl psql -U postgres -At -c "SELECT md5(string_agg(table_name||'.'||column_name||':'||data_type,',' ORDER BY table_name,ordinal_position)) FROM information_schema.columns WHERE table_schema='public';" +``` +Pass: **F1 == F2**, 125→125 tables, `db:migrate` applied nothing. ⚠️ Seed **before** migrate. +Do this in a **throwaway checkout** — do NOT commit the copy into the live `migrations/` path +until the separately-approved promotion (PR #247 governs). + +## Step 6 — Teardown + deliver to the replacement PR + +```powershell +docker rm -f zf-repl +Remove-Item -Recurse -Force packages/db-client/src/postgres/prod-baseline-v2, packages/db-client/drizzle.config.replace.ts -ErrorAction SilentlyContinue +``` +Commit to the replacement PR (this branch): the authoritative `0000_prod_baseline.sql` + +`meta/_journal.json` (staged under `docs/audit/phase3b-promotion/replacement/prod-baseline/`, +quarantined like PR #248's), the **drift inventory** from Step 3, and the filled Step 6 report: + +``` +REPLACEMENT BASELINE (PG18, real snapshot) — + engine ........... : 18.4 + tables ........... : 125 + columns .......... : 1649 (authoritative) + indexes / FKs .... : 402 / 266 + int4_ops fixed ... : yes/no + diff vs PR #248 .. : +__ cols / +__ idx / -__ fk (inventory attached) + ZERO_DDL (PG18) .. : F1==F2? yes/no ; db:migrate applied nothing? yes/no + RESULT ........... : PASS -> supersede PR #248 +``` + +## Decision + +- **PASS** → the snapshot-derived baseline supersedes PR #248; **close PR #248** in favor of + this replacement PR. B3 cleared. Proceed to B2 (restore/PITR), still separately approved. +- **FAIL** → iterate on the baseline; PR #248 stays superseded, neither adopted. + +## Guardrails + +No production deploy · migration execution against prod · schema application to prod · +backend wiring · feature-flag activation. Throwaway PG18 only; production never targeted. diff --git a/docs/audit/phase3b-promotion/replacement/README.md b/docs/audit/phase3b-promotion/replacement/README.md new file mode 100644 index 00000000..34aad235 --- /dev/null +++ b/docs/audit/phase3b-promotion/replacement/README.md @@ -0,0 +1,52 @@ +# Phase 3B — 0000_prod_baseline Replacement Plan (supersedes PR #248) + +> **PLAN / EVIDENCE — DOCS ONLY.** B3 verification proved PR #248's baseline (generated from +> `initDatabase()`) is **not faithful to production**. This package documents the confirmed +> drift and the plan to regenerate the authoritative baseline from the **operator's real +> Gate 1 snapshot on PG18**. **No production deploy, migration execution, schema application, +> backend wiring, or feature-flag activation.** PR #248 stays **unmerged / superseded**. + +## Why PR #248 is superseded + +B3 (PG18, real Gate 1 snapshot) vs PR #248 (initDatabase, PG16), measured with **identical +catalog queries** (`information_schema` / `pg_indexes`): + +| Metric | PR #248 (`initDatabase()`) | Real prod (Gate 1 / PG18) | Delta | +|---|---|---|---| +| Tables | 125 | 125 | 0 ✅ | +| **Columns** | **1595** | **1649** | **+54 (real)** | +| Indexes (`pg_indexes`) | 400 | 402 | +2 | +| FKs | 267 | 266 | −1 | +| Non-PK indexes | 275 | 277 | +2 | + +Applying all **24 hand-written migration `.sql` files** on top of `initDatabase()` moved +columns only **1595 → 1596** — so the migrations do **not** explain the gap. **~53 production +columns exist in no source of truth** (not `initDatabase()`, not the migration files) — plus ++2 indexes and a −1 FK. This is **undocumented production drift** (the core Phase-3 problem), +and it makes the `initDatabase()`-derived baseline unsafe to adopt. + +**Conclusion:** the authoritative `0000_prod_baseline` must be introspected from the **real +production schema** (the Gate 1 snapshot), not from `initDatabase()`. PR #248's candidate is +**suspect and replaced**. + +## Documents + +| Doc | Purpose | +|---|---| +| [`01_DRIFT_FINDING.md`](./01_DRIFT_FINDING.md) | Full evidence: apples-to-apples counts + migration test | +| [`02_REPLACEMENT_RUNBOOK.md`](./02_REPLACEMENT_RUNBOOK.md) | Operator steps to regenerate + diff + journal + zero-DDL on PG18 | + +## Status + +- **B3:** CONDITIONAL/PARTIAL PASS only — table count (125) verified on PG18; column/index/FK + drift **confirmed real**; ZERO-DDL migrate-skip **not yet proven** on PG18. +- **PR #248:** unmerged; to be **superseded** by the snapshot-derived baseline (this plan). +- **This is the replacement/superseding track** requested. The authoritative baseline + artifacts are produced by the operator on PG18 from the real snapshot (runbook doc 02) and + committed to the replacement PR. + +## Guardrails + +❌ No production deploy · migration execution · schema application · backend wiring · +feature-flag activation. ✅ Diagnostics ran on throwaway PG16 (torn down); production never +touched; live migrations path unchanged; `db:migrate` still no-op.