Feature: per-table description checkpointing — adding/changing one table shouldn't regenerate all descriptions

[Charbelkhayrallah]

Version: @kaelio/ktx@0.16.0. Related to (and best done after) the extractedAt resume-hash bug.

Motivation

Description enrichment caches at whole-batch granularity: the resume hash covers the entire snapshot's tables[], and the resume record is gated all-or-nothing on it. So changing enabled_tables — or any single table's columns — re-keys the batch and re-sends every table to the LLM, discarding expensive per-table descriptions that didn't change.

Verified (built from source, extractedAt bug already fixed so it's isolated): enabled_tables=4 → ingest, then add 2 tables → ingest regenerates all 6; the 4 unchanged tables are reworded, not reused. On a 300-table warehouse, adding one table means paying to re-describe 300.

This matters most on the rate-limited claude-code backend, where re-describing unchanged tables burns the Pro/Max window.

Current design (for context)

Code comments describe two layers: spec-19 = whole-stage inputHash gate; spec-20 = a per-table resume record. Per-table recovery already exists — generateDescriptions recovers tables present in the record and only enriches the remainder — but only while the whole-stage inputHash matches, i.e. resuming an interrupted run of the same table set. A table-set change flips the hash and discards the whole record. So per-table reuse across schema/scope changes is not supported today.

The opportunity: an intended seam already exists but is unwired

KtxDescriptionGenerator already has a complete per-table and per-column and per-connection cache seam — KtxDescriptionCachePort (description-generation.ts), with get-before / set-after around every LLM call (table 567/657, connection 949/997, column 1022/1103).

It has no production implementation — only a test fake — and is never injected (local-enrichment.ts constructs the generator with no cache). That idle seam also contradicts the repo's own docs/code-design.md:

MUST NOT add an optional dep-injection slot (deps.X ?? defaultX) unless at least one test exercises the production default.

This strongly suggests per-table caching was intended to flow through this port and was left unfinished.

Proposal

Implement and wire a production KtxDescriptionCachePort backed by the existing persistent ContentResultCache (the same (namespace, scopeKey, inputHash) store findCompletedStage uses):

• Key each table/column on its content — the referenced table/column's structure + llmIdentity — under a namespace like scan:description-table / scan:description-column.
• Inject it at the single generator construction site in local-enrichment.ts.

Result: every table/column description is content-addressed and reused across runs regardless of what else is in the batch — surviving scope changes, unrelated schema drift, and interruptions — in one place, at the seam already designed for it. Per-column coverage comes for free.

Design caveat to get right

The port's key methods currently receive only a KtxTableRef (catalog/db/name = identity), not the table's structure. A naive identity-keyed implementation would serve stale descriptions after a column change. The production implementation must derive a content key (hold the current snapshot; hash the referenced table/column's structure), or the port's key contract should be widened to pass structure. This is the one correctness-sensitive part.

Alternative (heavier)

Extend the spec-20 resume record to store a per-table content hash and recover per table (validated against the current structure). This works but adds a parallel mechanism in the resume/flush path; wiring the existing port is the more localized, intent-aligned option and also covers column descriptions.

---

Related bug: #347