Skip to content

Commit bf654cf

Browse files
authored
fix(migrations): restore 0180's bytes and freeze released migrations (#9420) (#9776)
#9420 updated a doc COMMENT inside migrations/0180_decision_ledger.sql. No DDL changed. It shipped in orb-v3.5.0. runSelfHostMigrations records a sha256 of each applied migration's FULL text and re-hashes on every boot, throwing selfhost_migration_content_drift on a mismatch. Comments are part of that text. So every deployment that had already applied 0180 would REFUSE TO BOOT on upgrade -- not degrade, fail to start, review pipeline down until a human restored the file. Confirmed against the live ORB: its _selfhost_migrations row for 0180 stores 0a95840f..., byte-identical to orb-v3.5.0-beta.10 and different from main. A full sweep of all 199 ledger rows found 0180 to be the only such drift. Restore 0180 verbatim. Nothing is lost: #9420 wrote the same prose into src/review/decision-record.ts's header, which is where it belongs and where it can change freely -- that header now says so, and says why the .sql must not. Add db:migrations:immutable:check: once a migration exists in a released orb-v tag its bytes are frozen, baselined at its FIRST release so a later tag that disagrees (as 3.5.0 does) cannot move the target and block the repair. The 35 files edited before this rule existed are re-frozen at current content rather than exempted, so the guard has no permanent holes.
1 parent 244b0e2 commit bf654cf

6 files changed

Lines changed: 311 additions & 12 deletions

File tree

migrations/0180_decision_ledger.sql

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,10 @@
66
-- Every persistDecisionRecord write appends (including latest-finalize-wins rewrites of the same record id
77
-- -- supersessions are deliberately VISIBLE history, not silent replacement).
88
--
9-
-- HONEST LIMIT (module header repeats this; see migrations/0195_decision_ledger_anchors.sql, #9267): a
10-
-- self-operated chain is tamper-EVIDENT against every actor except the operator, on its own. As of #9267, a
11-
-- scheduled job (src/review/ledger-anchor-scheduler.ts) additionally publishes a SIGNED, self-describing
12-
-- checkpoint of this chain's tip -- hourly, or every 256 new rows, whichever comes first -- to two places the
13-
-- operator does not control: a Sigstore Rekor transparency log and a git commit (cross-mirrored by GH Archive
14-
-- / Software Heritage the moment it's pushed). Rewriting history before the oldest still-referenced anchor
15-
-- now requires forging that signature or fabricating matching evidence at an external mirror too, not just
16-
-- editing this table. What remains open: the UNANCHORED TAIL since the last checkpoint is exactly as
17-
-- tamper-evident-only as before anchoring existed -- anchoring bounds how far back an undetected rewrite
18-
-- could reach, it does not make every row individually external-checkable in real time.
9+
-- HONEST LIMIT (module header repeats this): a self-operated chain is tamper-EVIDENT, not tamper-PROOF --
10+
-- the operator can still rewrite wholesale. External anchoring (signed checkpoints / witness cosigning) is
11+
-- the tracked follow-up once tenants exist, per the epic's sequencing. That gap does not reduce the value
12+
-- against every OTHER actor, or against accidental corruption.
1913
CREATE TABLE IF NOT EXISTS decision_ledger (
2014
seq INTEGER PRIMARY KEY, -- explicit, contiguous (verified); NOT autoincrement -- gaps are breaks
2115
record_id TEXT NOT NULL, -- decision_records.id at append time

package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@
5757
"control-plane:coverage": "node --experimental-strip-types scripts/control-plane-coverage.ts",
5858
"engine:coverage": "node --experimental-strip-types scripts/engine-coverage.ts",
5959
"db:migrations:check": "tsx scripts/check-migrations.ts",
60+
"db:migrations:immutable:check": "tsx scripts/check-released-migrations-immutable.ts",
6061
"db:schema-drift:check": "tsx scripts/check-schema-drift.ts",
6162
"actionlint": "node --experimental-strip-types scripts/actionlint.ts",
6263
"lint:composite-actions": "node --experimental-strip-types scripts/lint-composite-actions.ts",
@@ -131,7 +132,7 @@
131132
"test:smoke:browser:install": "playwright install chromium",
132133
"test:smoke:browser": "node --experimental-strip-types scripts/smoke-ui-browser.ts",
133134
"pretest:ci": "npm run check-node-version",
134-
"test:ci": "git diff --check && npm run actionlint && npm run lint:composite-actions && npm run db:migrations:check && npm run db:schema-drift:check && npm run selfhost:env-reference:check && npm run miner:env-reference:check && npm run selfhost:validate-observability && npm run cf-typegen:check && npm run build --workspace @loopover/engine && npm run build --workspace @loopover/discovery-index && npm run build:mcp && npm run build:miner && npm run build --workspace @loopover/ui-kit && npm run typecheck && npm run test:coverage && npm run test:engine-parity && npm run test:live-gate-parity && npm run test:driver-parity && npm run validate:mcp && npm run test --workspace @loopover/engine && npm run test:workers && npm run test:mcp-pack && npm run test:miner-pack && npm run test:engine-pack && npm run test:ui-kit-pack && npm run test:miner-deployment-docs-audit && npm run rees:test && npm run ui:openapi:check && npm run ui:version-audit && npm run docs:drift-check && npm run coverage-boltons:check && npm run import-specifiers:check && npm run ui-derived-types:check && npm run server-manifest:check && npm run dead-source-files:check && npm run regate-sort-key:check && npm run command-redelivery-guards:check && npm run dispatch-gate-reasons:check && npm run validate:no-hand-written-js && npm run replay-runner-manifest:check && npm run coco-dev-versions:check && npm run branding-drift:check && npm run manifest:drift-check && npm run engine-parity:drift-check && npm run engines-nvmrc:check && npm run release-manifest:sync:check && npm run release-linked-versions:check && npm run command-reference:check && npm run mcp:tool-reference:check && npm run mcp:client-config:check && npm run contract:api-schemas:check && npm run ui:lint && npm run ui:typecheck && npm run ui:test && npm run ui:build",
135+
"test:ci": "git diff --check && npm run actionlint && npm run lint:composite-actions && npm run db:migrations:check && npm run db:migrations:immutable:check && npm run db:schema-drift:check && npm run selfhost:env-reference:check && npm run miner:env-reference:check && npm run selfhost:validate-observability && npm run cf-typegen:check && npm run build --workspace @loopover/engine && npm run build --workspace @loopover/discovery-index && npm run build:mcp && npm run build:miner && npm run build --workspace @loopover/ui-kit && npm run typecheck && npm run test:coverage && npm run test:engine-parity && npm run test:live-gate-parity && npm run test:driver-parity && npm run validate:mcp && npm run test --workspace @loopover/engine && npm run test:workers && npm run test:mcp-pack && npm run test:miner-pack && npm run test:engine-pack && npm run test:ui-kit-pack && npm run test:miner-deployment-docs-audit && npm run rees:test && npm run ui:openapi:check && npm run ui:version-audit && npm run docs:drift-check && npm run coverage-boltons:check && npm run import-specifiers:check && npm run ui-derived-types:check && npm run server-manifest:check && npm run dead-source-files:check && npm run regate-sort-key:check && npm run command-redelivery-guards:check && npm run dispatch-gate-reasons:check && npm run validate:no-hand-written-js && npm run replay-runner-manifest:check && npm run coco-dev-versions:check && npm run branding-drift:check && npm run manifest:drift-check && npm run engine-parity:drift-check && npm run engines-nvmrc:check && npm run release-manifest:sync:check && npm run release-linked-versions:check && npm run command-reference:check && npm run mcp:tool-reference:check && npm run mcp:client-config:check && npm run contract:api-schemas:check && npm run ui:lint && npm run ui:typecheck && npm run ui:test && npm run ui:build",
135136
"test:release": "npm run test:ci && npm run changelog:check",
136137
"test:release:mcp": "npm run test:ci",
137138
"test:watch": "vitest",
Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
#!/usr/bin/env node
2+
// A migration that has shipped in a release is IMMUTABLE (#9420 regression).
3+
//
4+
// THE INCIDENT THIS EXISTS FOR: #9420 updated a doc COMMENT inside migrations/0180_decision_ledger.sql --
5+
// no DDL change at all -- and shipped it in orb-v3.5.0. But runSelfHostMigrations (src/selfhost/migrate.ts)
6+
// records a sha256 of each applied migration's FULL TEXT and, on every boot, re-hashes the on-disk file and
7+
// throws `selfhost_migration_content_drift` if it differs. Comments are part of that text. So the moment any
8+
// already-upgraded deployment pulled the new image it would refuse to start -- not degrade, not warn: fail
9+
// to boot, with the review pipeline down until a human restored the file. Nothing caught it: the existing
10+
// db:migrations:check guards NUMBERING (collisions, gaps, filenames), and git reports a clean one-file diff
11+
// because editing a file is not a conflict.
12+
//
13+
// The rule is therefore mechanical, and this is the check that enforces it: once a migration file exists in
14+
// any released `orb-v*` tag, its bytes may never change again. Not the DDL, not a typo, not a comment.
15+
// Forward-only means forward-only -- to change what a migration DID, add a new one; to change what it SAYS,
16+
// put the prose in the source module that reads the table (see src/review/decision-record.ts's header for
17+
// exactly this split).
18+
//
19+
// Deleting a released migration is likewise refused. migrate.ts tolerates a ledger row whose file has
20+
// vanished (it skips unknown names), but a fresh deployment would then build a different schema than every
21+
// existing one -- a silent divergence this check would rather stop at the PR.
22+
import { execFileSync } from "node:child_process";
23+
import { readdirSync } from "node:fs";
24+
import { join } from "node:path";
25+
import { MIGRATION_REBASELINE } from "./migration-rebaseline";
26+
27+
/** A released tag and the migration blobs it shipped, as `name -> blob sha`. */
28+
export type ReleasedTagManifest = { tag: string; files: ReadonlyMap<string, string> };
29+
30+
export type MigrationViolation = { file: string; tag: string; kind: "modified" | "deleted" };
31+
32+
/**
33+
* PURE core: any file whose blob sha differs from (or is missing versus) the one it was FIRST released with.
34+
*
35+
* The baseline is deliberately the EARLIEST tag that shipped each file, not every tag. A file's first
36+
* release is when it froze: that is the content the oldest deployments applied and recorded a hash for, and
37+
* they are both the most numerous and the ones with the most history to lose. Checking against every tag
38+
* would be wrong here for a concrete reason -- when a released migration HAS been mutated (the #9420
39+
* incident), the released tags themselves disagree with each other, so no content could satisfy all of them
40+
* and the check could never go green again, not even after the correct repair.
41+
*
42+
* `released` must be ordered oldest-first; first sighting of a file wins.
43+
*/
44+
export function findMutatedReleasedMigrations(
45+
released: readonly ReleasedTagManifest[],
46+
current: ReadonlyMap<string, string>,
47+
): MigrationViolation[] {
48+
const frozen = new Map<string, { tag: string; blob: string }>();
49+
for (const { tag, files } of released) {
50+
for (const [file, blob] of files) if (!frozen.has(file)) frozen.set(file, { tag, blob });
51+
}
52+
// Files edited before this guard existed are re-frozen at their current content instead of their first
53+
// release -- see migration-rebaseline.ts for why that is safe here and why the table must never grow.
54+
for (const [file, blob] of MIGRATION_REBASELINE) {
55+
const existing = frozen.get(file);
56+
if (existing) frozen.set(file, { tag: existing.tag, blob });
57+
}
58+
59+
const violations: MigrationViolation[] = [];
60+
for (const [file, { tag, blob }] of frozen) {
61+
const currentBlob = current.get(file);
62+
if (currentBlob === undefined) violations.push({ file, tag, kind: "deleted" });
63+
else if (currentBlob !== blob) violations.push({ file, tag, kind: "modified" });
64+
}
65+
return violations.sort((a, b) => a.file.localeCompare(b.file));
66+
}
67+
68+
function git(...args: string[]): string {
69+
return execFileSync("git", args, { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
70+
}
71+
72+
/** `migrations/*.sql` at a given rev, as `name -> blob sha`. */
73+
function migrationsAt(rev: string): Map<string, string> {
74+
const files = new Map<string, string>();
75+
for (const line of git("ls-tree", "-r", rev, "--", "migrations/").split("\n")) {
76+
// `<mode> blob <sha>\t<path>`
77+
const match = /^\d+ blob ([0-9a-f]+)\t(migrations\/.+\.sql)$/.exec(line);
78+
if (match?.[1] && match[2]) files.set(match[2].slice("migrations/".length), match[1]);
79+
}
80+
return files;
81+
}
82+
83+
/** Released ORB tags, oldest first, so the earliest tag to freeze a file is the one reported. */
84+
export function releasedOrbTags(): string[] {
85+
return git("tag", "-l", "orb-v*", "--sort=creatordate").split("\n").filter(Boolean);
86+
}
87+
88+
/**
89+
* The migrations as they exist ON DISK, hashed with git's own blob algorithm so they compare directly
90+
* against `ls-tree` output.
91+
*
92+
* Deliberately NOT `ls-tree HEAD`: that reads the committed tree and is blind to uncommitted edits, so the
93+
* check would go green locally on exactly the change it exists to reject and only fail later in CI. Reading
94+
* the working tree makes it usable as a pre-commit check and makes what it reports match what the author is
95+
* actually about to ship.
96+
*/
97+
function migrationsOnDisk(): Map<string, string> {
98+
const files = new Map<string, string>();
99+
const names = readdirSync("migrations").filter((name) => name.endsWith(".sql")).sort();
100+
if (names.length === 0) return files;
101+
// One batched hash-object call: 200 separate spawns is the difference between instant and noticeable.
102+
const hashes = git("hash-object", "--", ...names.map((name) => join("migrations", name))).split("\n").filter(Boolean);
103+
names.forEach((name, index) => {
104+
const hash = hashes[index];
105+
if (hash) files.set(name, hash);
106+
});
107+
return files;
108+
}
109+
110+
function main(): void {
111+
const tags = releasedOrbTags();
112+
if (tags.length === 0) {
113+
// A shallow clone or a fork with no tags cannot evaluate this rule. Say so rather than passing silently:
114+
// a check that quietly becomes a no-op is how the thing it guards comes back.
115+
console.error("released-migrations-immutable: no orb-v* tags visible — fetch tags (`git fetch --tags`) so this check can run.");
116+
process.exit(1);
117+
}
118+
const released = tags.map((tag) => ({ tag, files: migrationsAt(tag) }));
119+
const violations = findMutatedReleasedMigrations(released, migrationsOnDisk());
120+
121+
if (violations.length > 0) {
122+
console.error("A migration that already shipped in a release was changed. Released migrations are immutable:\n");
123+
for (const { file, tag, kind } of violations) {
124+
console.error(` migrations/${file}${kind} (first released in ${tag})`);
125+
}
126+
console.error(
127+
"\n Every deployment that already applied one of these recorded a sha256 of its FULL text (comments\n" +
128+
" included). src/selfhost/migrate.ts re-hashes on every boot and throws selfhost_migration_content_drift\n" +
129+
" on a mismatch, so shipping this would make every already-upgraded ORB FAIL TO BOOT.\n\n" +
130+
" To change what a migration DID: add a new migrations/NNNN_*.sql.\n" +
131+
" To change what it SAYS: put the prose in the source module that reads the table, not the .sql.\n" +
132+
" To undo an accidental edit: git checkout <tag> -- migrations/<file>",
133+
);
134+
process.exit(1);
135+
}
136+
const frozen = new Set(released.flatMap(({ files }) => [...files.keys()])).size;
137+
console.log(`released-migrations-immutable: OK — ${frozen} released migration(s) unchanged across ${tags.length} orb-v tag(s).`);
138+
}
139+
140+
if (process.argv[1]?.endsWith("check-released-migrations-immutable.ts")) main();

scripts/migration-rebaseline.ts

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
// Released migrations that were ALREADY edited before the immutability guard existed (#9420 fallout).
2+
//
3+
// check-released-migrations-immutable.ts normally freezes a migration at the content of the first release
4+
// that shipped it. These 35 files cannot use that baseline: each was edited at some point across the project's
5+
// history -- mostly in the orb-v0.1.0 / orb-v0.4.0 era -- long before the rule was written down or enforced.
6+
//
7+
// They are RE-FROZEN here at their current content rather than exempted. Exempting them would leave 35
8+
// permanent holes in a guard whose whole value is having none; re-baselining keeps every one of them under
9+
// the rule from here on, and costs only this generated table. Verified when this landed: none of these drifts
10+
// against the live fleet -- every edit predates the deployments that applied the file, so no running ORB
11+
// recorded the older hash. That is what makes re-freezing safe rather than merely convenient.
12+
//
13+
// NOTE 0180_decision_ledger.sql is deliberately NOT here. It is the file the #9420 incident actually broke,
14+
// and it was repaired by restoring its original bytes -- so its first-release baseline is correct and it
15+
// stays under the normal rule.
16+
//
17+
// DO NOT ADD ENTRIES. A new entry means someone edited a released migration and papered over it, which is
18+
// precisely the boot failure this guard exists to prevent. Add a new migrations/NNNN_*.sql instead, and put
19+
// prose in the source module that reads the table (see src/review/decision-record.ts's header).
20+
export const MIGRATION_REBASELINE: ReadonlyMap<string, string> = new Map([
21+
["0029_ai_review_provider_model.sql", "e7533a461476ea7fecebc33cc6a7889295add479"],
22+
["0034_slop_ai_advisory.sql", "ab23e28623a484c93f045bc752407fbab2044e98"],
23+
["0035_pull_request_slop_assessment.sql", "56d78b31ff660e3a99c92208fbd5d9ac94fb53da"],
24+
["0042_agent_autonomy.sql", "85c28edacc98c92cc625c6371f5a29327577ca58"],
25+
["0047_self_improve_tunables.sql", "af6347526cff858922f7adfa1c32141cfb0a954b"],
26+
["0049_review_audit_parity.sql", "ebdcbcc7b65dd4fb25d96c5792d790330ef358b6"],
27+
["0050_review_targets.sql", "4f75b205186fd7818d29eb7a11a97b3e995d303e"],
28+
["0051_repo_chunks.sql", "7e1405a4742ccc3380ea12e70b6f9ad53ec63006"],
29+
["0052_pr_merge_attempt_terminal.sql", "7dc06935c33848233c15ade0a9df0cebef379630"],
30+
["0056_orb_events.sql", "daec50cef4efe09fa810f771963dd5d78a9cf94c"],
31+
["0057_orb_installations.sql", "e7c96f4500ff3e0f1bdd7811db11c682ae13f263"],
32+
["0058_orb_signals.sql", "796cd598d416adeebd8e3f5c4f0b8b48247b6424"],
33+
["0060_orb_fleet_collector.sql", "47a4d7756a5bbd8ab68490a9d2262004f1982d45"],
34+
["0061_orb_instances.sql", "151081374a2f7e1bb4f8e97e2978c06f7ea1f9ff"],
35+
["0062_pr_last_regated_at.sql", "d3bd6140263884b61aed73d144fc2179bef74c90"],
36+
["0064_orb_webhook_events.sql", "b8e60650f6986c0b67bd10543785c771c0e3be31"],
37+
["0065_orb_github_installations.sql", "1607d818eca85ce8ec561f6d8504ef15c8d5aad7"],
38+
["0067_orb_pr_outcomes.sql", "89eee98d18e50cce3dd1b874f0a12fc92fb406aa"],
39+
["0068_orb_enrollments.sql", "976aea26f676c9fcbe876a9389e7ee45db9bf46f"],
40+
["0071_installations_app_id.sql", "698697c7609dfea8ff42aadca2341705d4fbf9a8"],
41+
["0072_contributor_blacklist.sql", "626b6580274e8633f4bf3e7bf9cea1929128aef2"],
42+
["0080_pr_last_published_surface_sha.sql", "4fcca720632b83fbfdb6c9a57f1db156103d11c6"],
43+
["0091_review_nag_cooldown.sql", "924cde088aff88d0fe2b0d2bf93a3c510ecc1f82"],
44+
["0097_command_rate_limit.sql", "284cc6fffd0c3d079d43ea3cd5b2149fa4da5396"],
45+
["0100_review_nag_monitored_mentions.sql", "793664ddeb6c859fbafae573cf1a89ff20c55283"],
46+
["0102_fix_linked_issue_gate_mode_default.sql", "893273a3e6afba06f65bdf185e09a93cfc8566cc"],
47+
["0113_review_evasion_protection.sql", "61e5a23fcc1e377cb0bd2b01e5a31300cf1e43f9"],
48+
["0116_regate_sweep_order_mode.sql", "f5824a872f616d2d990d5ff2a4b8896cdbab1644"],
49+
["0119_ai_slop_cache.sql", "3fe8ad831ae0754c9a441998912d603e482707af"],
50+
["0126_contributor_gate_history.sql", "d8c7714ed65ae0845fcdb62bb68a3c06d4fb9f5f"],
51+
["0127_agent_global_freeze_override.sql", "78169578f5b45a0e2dd225781b562ca325ac404a"],
52+
["0134_pr_last_backlog_convergence_regated_at.sql", "ea71b7236b303a6b21cac8b0684adc969f629d13"],
53+
["0137_predicted_gate_calls.sql", "5a35dfc8ae2ceb8711a639daf59f2a59d87ffe67"],
54+
["0140_ai_review_low_confidence_disposition.sql", "bbe9077f0244d2d850aa392594272781e34860bd"],
55+
["0143_repository_skip_automation_bot_authors.sql", "36c139f3be5b48c43d40506080e1d536750d408c"],
56+
]);

src/review/decision-record.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,12 @@
1212
// (#8838) consume — one schema, three consumers, so drift between "what we published" and "what we can
1313
// replay" is structurally impossible.
1414
//
15-
// HONEST LIMIT (#9122, mirrored from migrations/0180_decision_ledger.sql's own header): the hash-chained
15+
// HONEST LIMIT (#9122). This is the CURRENT statement of it, and the only one that gets updated:
16+
// migrations/0180_decision_ledger.sql carries an older, shorter version in its own header and must keep it
17+
// verbatim forever. That file has already been applied on running deployments, and runSelfHostMigrations
18+
// hashes every applied migration's FULL text (comments included) to detect post-apply edits -- so editing
19+
// its prose, even harmlessly, makes every already-upgraded ORB fail to boot. Documentation about this table
20+
// belongs here, where it can change freely. The hash-chained
1621
// ledger below makes this instance's history tamper-EVIDENT against every actor except an operator with
1722
// direct DB access, on its own — such an operator could still rewrite the chain wholesale (delete every row,
1823
// recompute a fresh one from genesis) and nothing INTERNAL to this table can detect that from first

0 commit comments

Comments
 (0)