Bug: `ktx ingest` regenerates every AI description on each scan — the scan timestamp `extractedAt` is inside the resume hash

[Charbelkhayrallah]

Version: @kaelio/ktx@0.16.0 (npm latest); main unchanged. Backend: claude-code, descriptions=haiku.

Summary

Re-running ktx ingest re-sends every table/column description to the LLM even when the schema is completely unchanged. Cross-run resume never hits, so on a large DB every run effectively "restarts from 0" and exhausts the rate-limit window. The cause is that a per-scan wall-clock timestamp is folded into the description resume hash.

Root cause

computeKtxDescriptionsStageHash hashes the whole snapshot, and the snapshot carries extractedAt, set fresh on every scan:

// packages/cli/src/context/scan/enrichment-state.ts
export function computeKtxDescriptionsStageHash(input) {
  return stableContentHash({ snapshot: input.snapshot, llmIdentity: input.llmIdentity });
}

Connectors set extractedAt: this.now().toISOString() on every introspection (e.g. connectors/postgres/connector.ts). Both persistent caches key on this hash, so a fresh extractedAt = new hash = full miss:

• SQLite stage cache — runEnrichmentStage → stateStore.findCompletedStage({ connectionId, stage, inputHash }). On a hit it skips the LLM entirely.
• File resume store — enrichment-progress/descriptions.json (local-enrichment-artifacts.ts).

Evidence it's the timestamp, not data drift

Two scans of the identical table set: every per-table staged snapshot and the FK graph were byte-identical; the only diff was connection.json's timestamp:

extractedAt: "2026-07-09T06:49:07.775Z"  →  "2026-07-09T07:11:45.930Z"

…and the descriptions hash moved 55abf581… → cac78360…, forcing full regeneration. The SQLite cache (local_content_results, namespace scan:descriptions) shows the two scans as two rows that never dedupe.

Reproduction (any warehouse)

1. ktx ingest <conn> — note the descriptions and enrichment-progress/descriptions.json inputHash.
2. ktx ingest <conn> again (fresh scan, unchanged schema). Observe: descriptions reworded, new inputHash, and the SQLite scan:descriptions cache gains a second row instead of hitting the first.

Expected: an unchanged schema reuses prior descriptions and makes no LLM calls.

Impact

Every re-scan re-sends all tables to the LLM. On a Pro/Max subscription this exhausts the window; with ingest.workUnits.failureMode: continue, rate-limited tables are then left with empty descriptions.ai rather than pausing. No ktx.yaml option or CLI flag avoids it (--stages descriptions still recomputes the hash).

Fix

Exclude extractedAt (and any per-scan metadata) from computeKtxDescriptionsStageHash — hash only schema-identifying content. This mirrors code you already have: stableLiveDatabaseHashContent in local-stage-ingest.ts already does delete stable.extractedAt before hashing connection.json for the ingest work-unit cache; the scan-enrichment stage hash just needs the same treatment (applied to all three stage hashes, since embeddings/relationships hash the snapshot too).

Verified locally (built from source): with the strip, a second ktx ingest on an unchanged schema makes zero description LLM calls, keeps a stable stage hash, and writes no new write/flush enrichment descriptions commits. A unit regression (two snapshots differing only in extractedAt must hash equal) fails on main and passes with the fix; a real schema change still re-keys.

Scope note

This fixes re-runs against an unchanged schema (the common case: interruptions, retries, refreshes). Adding/removing/changing a table still re-keys the whole batch and regenerates unchanged tables too — that's a separate, deeper gap (per-table checkpointing) tracked as a feature request.

[sricursion]

I'd like to claim this one.

I've verified the root cause on main: computeKtxDescriptionsStageHash hashes the full snapshot, which carries the per-scan extractedAt, so an unchanged schema re-keys the stage on every run — and the embeddings/relationships stage hashes have the same exposure. Plan: strip extractedAt from the snapshot before hashing in all three stage-hash functions in enrichment-state.ts via one shared helper, mirroring what stableLiveDatabaseHashContent in local-stage-ingest.ts already does for the ingest cache, plus a regression test that two snapshots differing only in extractedAt hash equal while a real schema change still re-keys. Per-table reuse across scope changes stays out of scope here (that's #348).