Structured-batch evidence recorded with a REPL-redefined validator can never be cleared (identity-based request keying)

[shanev]

Finding

src/droste/structured.py's structured-batch evidence tracking (_StructuredBatchEvidence) keys an unresolved request by (prompts, contexts, schema, validator_token), where validator_token is assigned by Python object identity (known is validator in _validator_token). This is intentional, documented safety behavior: "Keeping validator objects alive prevents identity reuse from turning a different validator into false completion evidence" — a different validator must not be able to silently satisfy evidence recorded against an earlier one.

The sharp edge: if calling code defines its validator function inline inside REPL/cell code that later gets re-executed (e.g., the RLM loop's normal error-repair path re-running a cell after it raises), the re-executed cell defines a new function object for the same logical validator. Even though the new call has identical prompts/contexts/schema and functionally identical validation logic, its validator_token differs from the original, so it is recorded as a distinct request. The original unresolved batch becomes permanently unreachable — no future call, however correct, can ever satisfy that exact key again — while each such re-execution can add further orphaned unresolved batches. minimum_exact_retry_calls (the count needed to clear all unresolved requests) only grows; no subcall budget, however large, resolves it once one such orphaned batch exists.

Discovered via a droste#81 benchmark pilot: benchmarks/live.py's OOLONG semantic guidance instructs the model to def validate_labels(...) inline in a cell and call llm_batch_json(..., validator=validate_labels), raising RuntimeError on any remaining errors. Any outer-loop retry after that raise redefines validate_labels, orphaning prior unresolved evidence. Trajectory evidence from one failed pilot task showed the validator redefined 7 times across 6 iterations, with minimum_exact_retry_calls climbing from ~60 (at a 50-subcall ceiling) to ~159 (at a 150-subcall ceiling) — i.e. it tracks whatever ceiling is available rather than converging, confirming this is a structural ratchet, not an undersized budget.

Scope

This is engine behavior working as specified for its own safety goal, not a droste bug to fix by relaxing the identity contract. Filing this so the sharp edge is documented and discoverable, not to propose weakening structured.py's guarantees.

Suggested resolution direction

Document the define-once/exact-replay pattern explicitly: callers using llm_batch_json with a validator across multiple attempts must define the validator once (e.g., guard with if 'validate_labels' not in globals():) and reuse the same object across retries — including outer-loop cell re-executions — rather than relying on raise-and-re-execute to "retry." Consider whether docs/ (prompt-pack or engine-usage guidance) should call this out explicitly, since it's easy to hit for any caller pattern that treats a validation failure as a whole-cell error rather than an in-cell bounded-replay loop.

Follow-up

The droste#81 benchmark guidance fix (rewriting _OOLONG_SEMANTIC_GUIDANCE to replay within one cell using a persistent validator) is being tracked as part of #81 itself, not here.