fix(databases): retry transient index-backfill errors in-process instead of parking

[heskew]

Summary

A secondary-index backfill pass (runIndexing) that failed with only transient RocksDB errors (ERR_BUSY / ERR_TRY_AGAIN — e.g. the write buffer filling under bulk-ingest load) would park the index: leaving isIndexing=true + indexingFailed=true and waiting for an operator restart to retry. This change retries the whole pass in-process from the pass's starting checkpoint with bounded exponential backoff (≤10 retries, capped 1s), re-reading every row. A transient error now self-heals without a restart.

A permanent (non-transient) error, or exhausting the retry budget, still parks exactly as before — so this is strictly an improvement over the prior behavior.

Resolves #1356 (part of #1354).

Where to look

• resources/databases.ts — runIndexing. Retry is implemented via bounded recursion: on a transient-only failed pass it resets each attribute.lastIndexedKey to the pass start, persists it, backs off, and re-enters with an empty indicesToRemove (drops already done) and retryAttempt + 1.
• Key correctness decision (worth a look): the retry resets to the pass start, not the intra-pass checkpoint. That checkpoint (lastIndexedKey = key, written every 100 rows) can advance past a put that later rejects asynchronously, so resuming from it could skip a failed row and leave a silent index gap. Re-reading from the pass start re-covers it (indexing is idempotent). A regression test asserts exactly this.
• isTransientIndexingError mirrors the ERR_BUSY/ERR_TRY_AGAIN classification already used in DatabaseTransaction's commit retry.

Open item for the reviewer

Cross-model review (Codex + Gemini) surfaced one pre-existing, out-of-scope point I did not change: for multi-value attributes only the last put() per record is tracked by when(), so a non-last put that rejects is an unhandled rejection. This predates this PR (the inline comment at the post-loop await lastResolution already notes it). Flagging in case we want a follow-up issue; left alone here to keep scope tight.

Docs

Operator-facing surface is a new [warn] log line when a backfill retries (…hit transient errors; retrying pass N/10…) plus the self-healing behavior. No API/config/schema change — I don't think this needs a docs PR, but happy to add a sentence to the indexing/migration operational notes if wanted.

Tests

unitTests/resources/schemaMigrationFragility.test.js (RocksDB-only): transient errors retry to clean completion (exercising both the synchronous-throw and async-reject branches), permanent errors still park + recover on a simulated restart, and — the important one — a row that failed before an intra-pass checkpoint is re-covered by the retry (no gap).

---

🤖 Generated with Claude Code (Opus 4.8)

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[claude]

Reviewed; no blockers found.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[kriszyp]

The retry logic here is sound, but #410 so radically reworks this area that I think this PR would be better stacked on top of that one rather than merged independently. Given the scope of overlap, this feels like 5.2 material — worth waiting for #410 to land first so this doesn't need a rebase or conflict resolution after the fact. Happy to revisit once #410 is in! 🙂

— Kris (via Claude Sonnet 4.6)

[heskew]

@kriszyp - Sounds good. I'll pull this back to a draft for now.