fix+feat: v0.37.0 — recovery completeness, enumeration truth, provider compat, engine correctness, de-ossification

CHANGELOG.md

@@ -6,6 +6,95 @@ follow semantic versioning once it reaches 1.0.
 
 ## [Unreleased]
 
+## [0.37.0] - 2026-07-17
+
+_Four-angle audit batch: crash-recovery completeness, agent enumeration truth,
+provider compatibility, engine correctness, de-ossification, and redaction
+depth. Every security-floor bound is untouched (one is tightened)._
+
+### Fixed
+
+- **An evidence import interrupted by a crash no longer wedges forever.** The
+  download runs in-process, so a hard kill mid-download left the row `importing`
+  — a state that could never get back to `confirmed` (re-run) or `planned`
+  (re-confirm). Startup reconciliation now fails orphaned `importing` imports
+  (and their files), mirroring what it already did for runs.
+- **`list_objects` no longer makes keys 501–1000 of a page unreachable.** The
+  per-call echo cap (500) sat below the S3 page size (1000) while `next_token`
+  advanced past the whole page — so any enumeration with `max_keys > 500`
+  silently lost the tail keys with no way to page back to them. The echo cap now
+  equals the page cap (a full 1000-key page is ~50 KB, inside the elastic
+  tool-output budget that still backstops it).
+- **The final answer is never truncated silently.** The one unmarked cut in the
+  codebase: the answer contract was hard-sliced at 48 000 chars with no marker —
+  in post-processing that promises "write out EVERY item". The cap is now
+  model-elastic (≥ 4 chars/token of the completion budget, so it can never cut
+  an answer the model was allowed to emit) and, when hit, appends an explicit
+  `[TRUNCATED …]` marker.
+- **`test_conditional_get` no longer misreports "object changed" on providers
+  that ignore `If-None-Match`.** Many S3-compatible providers return `200`
+  (ignoring the conditional header) instead of `304`; the tool mapped any `200`
+  to `etag_matches: false` even with an identical ETag. It now compares
+  quote-normalized ETags: equal on `200` → "unchanged + conditional requests
+  unsupported" (rule 18), different → genuinely changed.
+- **`list_multipart_uploads` works on prefix-scoped providers.** The wrapper
+  exposed no `prefix`, so any provider with `allowed_prefixes` always denied the
+  root listing, making the abandoned-upload cost diagnostic unreachable there.
+  It now takes a `prefix`, passed to both the scope check and the S3 `Prefix=`.
+- **`get_object_lock_status` no longer reads a malformed call as "no lock".**
+  The broad `InvalidRequest` code was blanket-mapped to "none"; only its
+  object-lock flavor ("Bucket is missing Object Lock Configuration") means that.
+  Other `InvalidRequest`s now surface as errors instead of "cleanly deletable".
+- **Inventory engine:** no more `Storage-class skew: 'None' covers 100%` finding
+  when the inventory simply has no storage_class column (the same null-group
+  guard hot-key/hot-prefix already had); `average_object_size` uses floor
+  division so multi-PB totals keep int64 precision above 2^53.
+- **DuckDB layer:** a read-only open of a missing analytical DB is a clean
+  "nothing imported yet" error instead of creating a stray empty `.duckdb` via a
+  writable fallback; writer-side lock contention now gets the same friendly
+  retryable message readers already had.
+- **`session_datasets` dedupe matches NULL filenames** (`IS`, not `=`), so a
+  re-uploaded nameless file can't create two rows pointing at one on-disk path.
+
+### Security
+
+- **A pasted bare AWS access-key/secret-key PAIR is now fully scrubbed.** The
+  secret-key redaction rule is label-anchored (so bucket/object names aren't
+  blanket-mangled), which let a bare 40-char SK pasted alongside its `AKIA…` key
+  id survive redaction, be persisted, and re-enter the next turn's prompt. A
+  narrow rule now masks bare 40-char base64 tokens ONLY when the text also
+  carries an AWS access-key-ID shape — ordinary 40-char strings without that
+  hint remain untouched.
+- **`session_messages` JSON columns (`tool_activity`, `grounding`,
+  `proposed_actions`) pass through `redact()` at the persistence boundary**,
+  like every sibling repository — defense in depth for rule 14; the agent
+  runtime still sanitizes upstream.
+
+### Changed (de-ossification — no security-floor change)
+
+- **The completion budget's only upper bound is the model's real provider
+  max-output.** The module-wide 32 768 ceiling is gone: the per-model clamp
+  already existed, so the ceiling only ever bit models whose real output cap is
+  higher (claude-3-7 / gemini-2.5 at 64k, o-series at 100k) — starving long
+  enumerations on exactly the models that could hold them.
+- **The survey/config-review summary echoed to the agent scales with the model
+  window** (floor 2000 chars, ceiling 16k) instead of a flat 2000.
+- **The deterministic session summary scales too:** the persisted store holds up
+  to 200 facts/findings (was 50) and the context echo is model-elastic (floor
+  50), matching the agent-memory de-ossification; the human-readable digest
+  stays at 50 entries with an explicit "+N more" note.
+- **Size labels are binary to match the binary math** (KiB/MiB/GiB, thresholds
+  like `<4KiB`/`128KiB-1MiB`): the divisors were always 1024-based; only the
+  labels said KB/MB.
+- Docs: note that gated larger context windows (e.g. Claude 1M beta) should be
+  declared via the model provider's explicit `context_window` override.
+
+### Tests
+
+- 20 new regression tests (`test_v0370_fixes.py`) covering every item above;
+  existing tests updated where behavior intentionally changed (full-page echo,
+  provider-cap-only completion budget, object-lock message flavor).
+
 ## [0.36.0] - 2026-07-17
 
 _Migration crash-recovery: a partially-applied table-rebuild no longer wedges the

docs/tools.md

@@ -372,7 +372,11 @@ chars as a **hard floor**. A 128k/200k-context model is unchanged; a 1M-context
 model gets a proportionally deeper turn. The window comes from a built-in
 model→window table, but a model provider can carry an explicit `context_window`
 (tokens) that overrides it — so a newly-shipped large-context model isn't
-throttled to the default. The same window also scales the thread-replay caps
+throttled to the default. The table is deliberately conservative where a
+family's real window is gated: e.g. Claude models map to 200k (the GA default)
+even though some expose a 1M window behind a beta/tier opt-in — if your account
+has the larger window enabled, declare it via `context_window` on the model
+provider rather than expecting the table to assume it. The same window also scales the thread-replay caps
 (how many prior messages / chars the agent re-sees), floored at the historical
 values and capped. The completion (`max_tokens`) budget is additionally clamped
 to each model's real provider max-output, so we never send a value the provider

sidecar/app/agent_runtime/model_budget.py

@@ -69,7 +69,6 @@
 # window's fair share — no shipping model exceeds that usefully today).
 TOOL_OUTPUT_CHARS_CEILING = 2_000_000
 COMPLETION_TOKENS_FLOOR = 16_384        # was session_agent._MAX_COMPLETION_TOKENS
-COMPLETION_TOKENS_CEILING = 32_768      # stay under provider max-output caps
 
 
 def context_window(model: str | None, explicit: int | None = None) -> int:
@@ -116,13 +115,19 @@ def tool_output_char_budget(model: str | None, explicit_window: int | None = Non
 def completion_token_budget(model: str | None, explicit_window: int | None = None,
                             explicit_max: int | None = None) -> int:
     """Completion (max_tokens) budget: raised only where the window clearly
-    supports it, floored at the historical value, capped by the module ceiling AND
-    by the model's real provider max-output so we never trigger a 400.
+    supports it, floored at the historical value, and capped by the model's real
+    provider max-output so we never trigger a 400.
+
+    The per-model max-output clamp is the SOLE upper bound — there is no second
+    module-wide ceiling. (There was: 32_768, and since the provider clamp already
+    existed it only ever bit DOWNWARD on models whose real output ceiling is
+    higher — claude-3-7 / gemini-2.5 at 64k, o-series at 100k — starving long
+    enumeration answers on exactly the models that could hold them, worst on
+    reasoning models whose thinking spends part of the same budget.)
 
     The provider cap is applied only when it's *below* the floor for a genuinely
     small-output model — the floor otherwise wins (an existing deployment is
     unchanged), but a 4k-output model like gpt-4-turbo is clamped down to 4096
     rather than being handed the 16384 floor it would reject."""
-    scaled = max(COMPLETION_TOKENS_FLOOR,
-                 min(context_window(model, explicit_window) // 8, COMPLETION_TOKENS_CEILING))
+    scaled = max(COMPLETION_TOKENS_FLOOR, context_window(model, explicit_window) // 8)
     return min(scaled, max_output_tokens(model, explicit_max))

sidecar/app/agent_runtime/session_action_tools.py

@@ -40,7 +40,13 @@
     "account_discovery": "Discover account-level buckets and evidence sources.",
 }
 
+# FLOOR on the survey/review final_summary echoed to the model, scaled with the
+# model window in build() (same de-ossification as agent memory / thread replay):
+# the summary is an already-sanitized deterministic aggregate — no raw rows — so
+# clipping it to a flat 2000 chars on a large-window model just made the agent
+# narrate a large account from a truncated summary.
 _MAX_SUMMARY = 2000
+_MAX_SUMMARY_CEIL = 16000
 
 
 def _err(msg: str) -> str:
@@ -110,15 +116,16 @@ def _go() -> None:
     return run_id
 
 
-def _run_result(conn: sqlite3.Connection, run_id: str) -> dict[str, Any]:
+def _run_result(conn: sqlite3.Connection, run_id: str,
+                summary_cap: int = _MAX_SUMMARY) -> dict[str, Any]:
     row = runs_repo.get_row(conn, run_id)
     if row is None:
         return {"run_id": run_id, "status": "unknown"}
     summary = row["final_summary"] or ""
     result: dict[str, Any] = {
         "run_id": run_id,
         "status": row["status"],
-        "final_summary": redact_text(str(summary))[:_MAX_SUMMARY],
+        "final_summary": redact_text(str(summary))[:summary_cap],
     }
     if row["status"] in ("pending", "running"):
         # Hit the wall-clock timeout: the run is still going in the background.
@@ -140,6 +147,8 @@ def build(
     session_id: str | None = None,
     turn_id: str | None = None,
     cancel_event: Any = None,
+    model: str | None = None,
+    explicit_window: int | None = None,
 ) -> list[Any]:
     """Build the agent's read-only survey/review tools (always available).
 
@@ -154,6 +163,13 @@ def build(
     ``cancel_event`` lets the 180 s inline-run wait return early when the user
     stops the turn.
     """
+    from . import model_budget
+
+    # Elastic summary echo: floor at _MAX_SUMMARY (128k/200k windows unchanged),
+    # scaled with the window like agent memory / thread replay.
+    window = model_budget.context_window(model, explicit_window)
+    summary_cap = min(_MAX_SUMMARY_CEIL, _MAX_SUMMARY * max(1, window // 128_000))
+
     def provider(provider_id: str):
         return cloud_repo.get(conn, provider_id)
 
@@ -196,7 +212,7 @@ def review_bucket_config(provider_id: str, bucket: str) -> str:
                              user_prompt=_DEFAULT_PROMPTS["bucket_config_review"], session_id=session_id)
             run_id = _execute_run(conn, body, turn_id, f"bucket_config_review:{provider_id}:{bucket}",
                                   cancel_event=cancel_event)
-            result = _run_result(conn, run_id)
+            result = _run_result(conn, run_id, summary_cap)
         except Exception as exc:  # noqa: BLE001 — a tool returns an error string, never raises
             return _err(f"review_bucket_config failed: {exc}")
         note("review_bucket_config", bucket, result["status"])
@@ -219,7 +235,7 @@ def survey_account(provider_id: str, max_buckets: int = 0) -> str:
                              session_id=session_id, max_buckets=mb)
             run_id = _execute_run(conn, body, turn_id, f"account_discovery:{provider_id}",
                                   cancel_event=cancel_event)
-            result = _run_result(conn, run_id)
+            result = _run_result(conn, run_id, summary_cap)
             profile = account_repo.get_profile(conn, run_id)
         except Exception as exc:  # noqa: BLE001 — a tool returns an error string, never raises
             return _err(f"survey_account failed: {exc}")
@@ -244,7 +260,7 @@ def read_run_result(run_id: str, wait_seconds: int = 0) -> str:
         linked = {r["run_id"] for r in sessions_repo.list_runs(conn, session_id)} if session_id else set()
         if run_id not in linked:
             return _err("Unknown run_id for this session. Only runs in this session can be read.")
-        result = _run_result(conn, run_id)
+        result = _run_result(conn, run_id, summary_cap)
         # Bounded in-turn wait: poll until the run leaves pending/running or the
         # budget elapses. This whole turn already runs on a dedicated worker
         # thread (boto3 tools block it by design), so sleeping here stalls only
@@ -255,7 +271,7 @@ def read_run_result(run_id: str, wait_seconds: int = 0) -> str:
                 break  # user stopped the turn — stop waiting on the background run
             _time.sleep(1.0)
             conn.commit()  # end the read snapshot so run_sync's writes are visible
-            result = _run_result(conn, run_id)
+            result = _run_result(conn, run_id, summary_cap)
         audit.record(conn, "session.read_run_result",
                      {"session_id": session_id, "run_id": run_id, "status": result["status"]},
                      run_id=run_id)