fix(backfill): gate the open-data crawl on its own segments, not a clock every job bumps

[JSONbored]

Summary

recent_merged_pull_requests stopped being written on 2026-07-09T23:26Z. The writer was never removed, flag-gated, or throwing — the entire open-data backfill fan-out (labels, open_issues, open_pull_requests, recent_merged_pull_requests) has been skipped on every 30-minute cron tick since, because #4529's freshness gate reads a clock that unrelated jobs keep rewriting.

enqueueRepositoryOpenDataBackfill measured freshness on repo_sync_state.lastCompletedAt. refreshRepoSyncStateFromSegments rewrites that repo-wide column at the tail of every segment write path, including two that dispatch no open-data work and run far more often than FRESH_SYNC_MS: the ~2-minute re-gate sweep's own force: true open_pull_requests refresh (src/queue/processors.ts:1094) and the backfill-pr-details follow-on it enqueues. The clock never aged past six hours, so the gate never opened.

The timing is exact: a6fb46ab (#4529) landed 2026-07-09T23:27:09Z; the last open-data crawl completed 2026-07-09T23:26:01Z.

Only recent_merged_pull_requests showed the damage — it is the one open-data table with no webhook or sweep writer to mask a dead crawl. It has been serving a three-week-old window to the copycat containment engine (src/queue/copycat-detection.ts:99, wired into the live gate by #5999), the repo culture profile (worse than stale: its cache-invalidation signal is countRecentMergedPullRequests, so a frozen count means the profile never regenerates at all), the maintainer recap, six services, seven API routes, and six MCP tools.

The table is not superseded by pull_requests: nine of twelve repos in it have zero merged rows in pull_requests, and for the three installed repos it holds ~2.5x the pre-07-09 merged history, 5,383 rows of it with the hydrated changed_files_json the copycat engine matches on. Full evidence in #10193.

What changed

• Freshness now reads the oldest completion across the four segments the fan-out actually dispatches, treating a never-run segment as infinitely old. This measures attempts, not successes, so fix(backfill): scheduled per-repo backfill ignores freshness/error-backoff, re-syncs every repo every 30 min forever #4497's throttle is fully intact — a repo that genuinely completed a crawl still backs off for six hours, and a segment that came back waiting_rate_limit minutes ago is not re-attempted. The error-backoff window still reads lastCompletedAt; a repo-level error state is repo-wide by nature.
• The skip was silent — returned only as a warnings[] string that the cron caller (src/queue/job-dispatch.ts:202) discards, which is why three weeks of a starved crawl left no log, metric, or audit trail and was found only by noticing the table had stopped growing. It now emits loopover_github_open_data_backfill_skipped_total (labelled by reason) and a log line carrying the measured crawl age.
• Removed listContributorRecentMergedPullRequests, a reader with no production callers.

Recovering the missed window — and a 30-day clock

Correcting an earlier reading of mine: the first unblocked run does not close the whole gap. A scheduled run fetches only the top SEGMENT_PAGE_BUDGET pages of /pulls?state=closed&sort=updated&direction=desc — 2 pages (200 PRs) on light, 10 (1000) on the 6-hourly full window — and recent_merged_pull_requests is deliberately excluded from the resume re-send at src/github/backfill.ts:625, while progressiveHistory maps "complete + hasMore" to sampled rather than running. So a scheduled run never paginates deeper than its page budget. The recent end of the gap self-heals immediately; the deeper 2026-07-09 → mid-July portion needs one forced full/resume crawl via the admin segment endpoint.

That matters more than it normally would, because the table is on a deletion clock. src/db/retention.ts:97 prunes it at 30 days on updated_at, and nothing has refreshed updated_at since 2026-07-11T06:25Z — so on the current trajectory the whole table (6,927 rows) ages out by roughly 2026-08-10. The retention policy's own rationale is that these mirrors are "re-fetched from GitHub on the next sync of the PR that needs them"; that assumption is precisely what this bug broke, which turns retention from a bound on growth into an unopposed deletion path. Restoring the writer is what stops the countdown, since an upsert refreshes updated_at.

Rows for the nine uninstalled repos in the table will still age out — they are not dispatched at all — which is consistent with the table's design as a rolling mirror rather than an evidentiary trail.

Scope

• Narrow, one coherent change, inside wantedPaths
• No site/, CNAME, **/lovable/**, no changelog edit
• No secrets or private scoring terms

Validation

• npm run typecheck
• npm run test:changed — 13,075 passed. One pre-existing failure (test/integration/api.test.ts > serves private MCP tool listing and tool calls) reproduces identically on a clean tree at this base and is unrelated to this diff.
• npx vitest run test/unit/backfill.test.ts test/unit/data-spine.test.ts — 166 passed
• Metric-registry suites: selfhost-metrics, alerts-metric-name-references, docs-selfhost-troubleshooting-metric-names, env-example-metric-names
• npm run selfhost:validate-observability, npm run docs:drift-check
• Per-line/per-branch coverage audit over the diff's own line ranges in src/github/backfill.ts: every changed statement and both arms of every changed branch are covered
• git diff --check clean

Tests added: a regression test reproducing the exact production shape (segments three weeks old, sync state seconds old ⇒ still syncs, and specifically re-dispatches recent_merged_pull_requests); invariants for a missing segment row, an unparseable segment timestamp, the full status table the fresh-success window covers (success/partial/capped) plus one outside it, both error-backoff edges, and the skip's log/reason on both paths. Three existing #4497 tests now seed segment rows alongside the sync state — in production a success state only exists because those rows were rolled up into it, so this makes them model reality rather than relaxing them.

Safety

• No auth/CORS surface touched
• No secrets, tokens, wallets, hotkeys, trust scores, or reward values in code, tests, or this description
• Behaviour is strictly less restrictive on a starved crawl and unchanged on a genuinely fresh one

Closes #10193.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[loopover-orb]

Important

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏳ LoopOver is waiting…

LoopOver has seen this pull request and is waiting on CI checks to finish before reviewing it. This comment will update once the review runs.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed · 🟨 Waiting}

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
25858	1	25857	21

View the top 1 failed test(s) by shortest run time

test/integration/api.test.ts > api routes > serves private MCP tool listing and tool calls

Stack Traces | 0.502s run time

AssertionError: expected 500 to be 200 // Object.is equality

- Expected
+ Received

- 200
+ 500

 ❯ test/integration/api.test.ts:5098:44

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}