fix+feat: v0.34.0 — engine correctness, lifecycle robustness, agent de-ossification, model config

CHANGELOG.md

@@ -6,6 +6,88 @@ follow semantic versioning once it reaches 1.0.
 
 ## [Unreleased]
 
+## [0.34.0] - 2026-07-16
+
+_Analysis-engine correctness, lifecycle robustness, and agent de-ossification. A
+three-angle audit (agent ossification, DuckDB engine math, state-machine
+lifecycle) plus the reviewed third-party findings — bugs that misreported numbers,
+wedged state forever, or needlessly boxed the agent in._
+
+### Fixed — analysis engine correctness
+
+- **Status codes and sizes are integers, not floats.** A single unparsed log line
+  (or a missing inventory size) coerced the whole numeric column to float64 →
+  DuckDB DOUBLE, so reports showed status codes as `404.0` and object sizes above
+  2^53 lost precision (and `total_size` accumulated in DOUBLE). The integer columns
+  are now built as nullable Int64, preserving both the label and the value.
+- **Access-log error rates are "of requests", not "of lines".** `error_rate_4xx/5xx`
+  and the 206/404/403 shares divided by every ingested line, including
+  text-fallback rows with no status — diluting the rate by the unparsed fraction
+  and silently under-reporting errors across the whole [0.5, 1.0) parsed band the
+  truth guard allows. They now divide by the count of rows that actually parsed a
+  status code.
+- **Inventory average/small-object figures use consistent denominators.**
+  `average_object_size` is now `total_size / object_count` (so the displayed
+  total/count/avg triple reconciles), and `small_object_ratio` is computed over
+  objects that HAVE a size (not diluted by null-size rows).
+- **Object age bucketing is timezone-independent.** The DuckDB connection now pins
+  `TimeZone='UTC'`, so `datediff` against `current_timestamp` no longer lands
+  objects in the adjacent age bucket when the sidecar runs outside UTC.
+
+### Fixed — lifecycle robustness
+
+- **A failed dataset-persist no longer wedges an evidence import in `importing`
+  forever.** The post-download persistence ran outside any try/except (and had no
+  startup reconciler, unlike a run), so a DB error there left an import that could
+  never be re-confirmed or re-run. It now reverts to `failed` and cleans up.
+- **The blocking turn path resolves its turn handle on a persist failure**, like
+  the streaming worker already did — otherwise a commit error left the handle
+  un-done and non-evictable, stalling every subsequent turn in that session for
+  120s until eviction.
+- **The loser of a concurrent evidence-import claim fails its orphan run** instead
+  of leaving a permanent session-unlinked `pending` row.
+- **A `report_ready` publish failure can't downgrade a completed run to `failed`**,
+  and the executor's failure branch refuses to overwrite a terminal state.
+
+### Fixed — model provider & budget (reviewed findings)
+
+- **The provider test no longer gives false green/red.** A 404/405 on `/models`
+  (common on minimal proxies) was reported as a confident pass even though the key
+  was never verified; and a valid empty `base_url` (which uses the OpenAI default,
+  exactly like the real client) was flagged "configuration incomplete." The test
+  now reports "reachable, key unverified" as a caution and treats `base_url` as
+  optional.
+- **The tool-output budget is a hard cap, not a soft one.** A single tool return
+  was only counted AFTER it landed in context, so one large result could blow past
+  the per-turn budget. An output that would exceed the remaining budget is now
+  withheld with a valid JSON "too large — narrow it" envelope.
+
+### Changed — de-ossification (don't box the agent in)
+
+- **Custom aggregation gained real expressiveness** without loosening the
+  whitelist-and-bound-params floor: a SECOND group-by dimension (cross-tabs like
+  "403s per masked-IP per day"), `day`/`weekday` time buckets, and
+  `distinct_ips`/`distinct_keys`/`p99`/`min`-`max`-bytes metrics — all fixed SQL
+  fragments, zero raw-row exposure. Top-N now has a deterministic tiebreaker.
+- **Operator-declarable max output tokens** (`max_output_tokens` on a model
+  provider) clamps the completion budget, so a third-party/unknown model whose real
+  cap is lower doesn't get a `max_tokens` its endpoint 400s on — symmetric with the
+  existing context-window override.
+- **Agent working-memory recall now scales with the model window** (floored at 50),
+  like thread replay already did — a long investigation on a large-context model no
+  longer has its own recorded facts/findings clipped first. The per-turn skill-load
+  guard was raised (the elastic tool-output budget is the real bound), and the
+  survey bucket ceiling raised to 2000.
+- **The agent is reminded tool-result text is untrusted** — carried from v0.33; no
+  change here.
+
+### Fixed — frontend
+
+- **The 409 blocking-fallback captures its "answer baseline" before the turn
+  starts**, not from a GET issued after the 409 — closing a race where an answer
+  persisted in that window poisoned the baseline and the UI hung "running" for ~2.5
+  minutes before recovering.
+
 ## [0.33.0] - 2026-07-16
 
 _S3-compatible provider correctness round. The product's core promise — works

frontend/src/api.ts

@@ -111,6 +111,9 @@ export interface ModelProviderInput {
   /** Optional explicit context window (tokens). Overrides the built-in model
    * table so a new large-context model isn't throttled to the default. */
   context_window?: number | null;
+  /** Optional explicit max output tokens. Clamps the completion budget so a
+   * third-party/unknown model whose real cap is lower doesn't 400. */
+  max_output_tokens?: number | null;
 }
 
 export const listModelProviders = () =>

frontend/src/hooks/useTurnRunner.ts

@@ -202,11 +202,15 @@ export function useTurnRunner(opts: {
   // "ok" once the persisted answer is visible, or "inprogress" if it gives up —
   // in which case the caller keeps the pending bubble rather than dropping the
   // user's message (F4).
-  const waitForPersistedTurn = async (id: string): Promise<Outcome> => {
-    // The assistant answer for this turn is a NEW assistant message. The 409
-    // guarantees it isn't persisted yet, so the first successful fetch is a safe
-    // baseline; later fetches detect a new assistant id.
-    let baseline: Set<string> | null = null;
+  const waitForPersistedTurn = async (
+    id: string,
+    baselinePromise?: Promise<Set<string> | null>,
+  ): Promise<Outcome> => {
+    // The assistant answer for this turn is a NEW assistant message. Prefer the
+    // baseline captured BEFORE the turn started (baselinePromise) — it can't
+    // include this turn's answer, so it's race-free. Only if that snapshot failed
+    // do we fall back to capturing from the first successful fetch here.
+    let baseline: Set<string> | null = baselinePromise ? await baselinePromise : null;
     const captureOrDetect = (d: SessionDetail): boolean => {
       const asstIds = d.messages.filter((m) => m.role === "assistant").map((m) => m.id);
       if (baseline === null) {
@@ -215,10 +219,12 @@ export function useTurnRunner(opts: {
       }
       return asstIds.some((mid) => !baseline!.has(mid));
     };
-    try {
-      captureOrDetect(await getSession(id));
-    } catch {
-      /* the loop retries the fetch; baseline stays null until one succeeds */
+    if (baseline === null) {
+      try {
+        captureOrDetect(await getSession(id));
+      } catch {
+        /* the loop retries the fetch; baseline stays null until one succeeds */
+      }
     }
     // Bounded backoff aligned with the server's own turn budget (its blocking
     // wait is ~150 s), then give up polling — but never drop the message.
@@ -263,6 +269,15 @@ export function useTurnRunner(opts: {
       return;
     }
     const turnId = newTurnId();
+    // Snapshot the assistant-message ids BEFORE the turn runs (in parallel with
+    // the stream, so no added latency). The 409 blocking-fallback uses this as its
+    // "which assistant messages predate this turn" baseline. Capturing it here —
+    // rather than from a GET issued AFTER the 409 — closes a race where the worker
+    // persisted the answer in the gap after the 409 but before that GET, poisoning
+    // the baseline so the new answer was never detected and the UI hung (P2b).
+    const preTurnAsstIds: Promise<Set<string> | null> = getSession(id)
+      .then((d) => new Set(d.messages.filter((m) => m.role === "assistant").map((m) => m.id)))
+      .catch(() => null);
     patchSessionRun(id, {
       busy: true, error: null, needKey: false, pending: q,
       streamText: null, streamTools: [], stopped: false, stalled: false,
@@ -328,7 +343,7 @@ export function useTurnRunner(opts: {
         // The turn is still running server-side (nothing persisted yet). Poll
         // until this turn's assistant answer is actually persisted, then clear
         // the pending bubble — never on a fixed timer (F4).
-        outcome = await waitForPersistedTurn(id);
+        outcome = await waitForPersistedTurn(id, preTurnAsstIds);
         if (outcome === "inprogress") {
           // Gave up waiting, but the turn may still be running (its answer may
           // already be persisted server-side). Keep the pending bubble — the

frontend/src/i18n.tsx

@@ -280,6 +280,8 @@ const en: Dict = {
   "prov.fApiKey": "API key",
   "prov.fContextWindow": "Context window (tokens, optional)",
   "prov.hintContextWindow": "Overrides the built-in model table for the agent's depth budgets; leave empty to infer from the model name.",
+  "prov.fMaxOutput": "Max output tokens (optional)",
+  "prov.hintMaxOutput": "Clamps the completion budget so a third-party or unknown model whose real cap is lower doesn't reject the request; leave empty to infer from the model name.",
   "prov.fProvider": "Provider",
   "prov.fEndpoint": "Endpoint URL",
   "prov.fRegion": "Region",
@@ -310,6 +312,7 @@ const en: Dict = {
   "prov.confirmDeleteCloud": "Delete cloud provider \"{name}\"?",
   "prov.testOk": "OK",
   "prov.testIncomplete": "incomplete",
+  "prov.testUnverified": "reachable (key unverified)",
   // command palette
   "palette.placeholder": "Search chats or run a command…",
   "palette.newChat": "New chat",
@@ -588,6 +591,8 @@ const zh: Dict = {
   "prov.fApiKey": "API Key",
   "prov.fContextWindow": "上下文窗口（tokens，可选）",
   "prov.hintContextWindow": "覆盖内置模型表，用于智能体深度预算；留空则按模型名推断。",
+  "prov.fMaxOutput": "最大输出 tokens（可选）",
+  "prov.hintMaxOutput": "限制补全预算，避免第三方或未知模型因真实上限更低而拒绝请求；留空则按模型名推断。",
   "prov.fProvider": "提供商",
   "prov.fEndpoint": "Endpoint URL",
   "prov.fRegion": "区域",
@@ -618,6 +623,7 @@ const zh: Dict = {
   "prov.confirmDeleteCloud": "删除云存储提供商“{name}”？",
   "prov.testOk": "正常",
   "prov.testIncomplete": "不完整",
+  "prov.testUnverified": "可达（密钥未验证）",
   "palette.placeholder": "搜索对话或运行命令…",
   "palette.newChat": "新对话",
   "palette.settings": "打开设置",

frontend/src/types.ts

@@ -8,6 +8,8 @@ export interface ModelProvider {
   has_api_key: boolean;
   /** Optional explicit context window (tokens); overrides the built-in model table for the agent's depth budgets. */
   context_window: number | null;
+  /** Optional explicit max output tokens; clamps the completion budget so a lower-cap endpoint doesn't 400. */
+  max_output_tokens: number | null;
   /** True for the provider the agent uses (explicitly activated; otherwise the oldest is the implicit default). */
   active: boolean;
   created_at: string;
@@ -41,6 +43,8 @@ export interface ModelProviderTestResult {
   ok: boolean;
   checks: Record<string, boolean>;
   detail: string;
+  /** true=key accepted, false=key rejected, null=reached but key unverified. */
+  api_key_verified?: boolean | null;
 }
 
 // --- S3 tool results (Phase 03) ---

frontend/src/views/ProvidersView.tsx

@@ -59,6 +59,7 @@ const emptyModelForm: ModelProviderInput = {
   model: "",
   api_key: "",
   context_window: null,
+  max_output_tokens: null,
 };
 
 function ModelProvidersPanel() {
@@ -90,6 +91,7 @@ function ModelProvidersPanel() {
       model: p.model ?? "",
       api_key: "", // never prefill secrets
       context_window: p.context_window ?? null,
+      max_output_tokens: p.max_output_tokens ?? null,
     });
     setEditing(p);
     setCreating(false);
@@ -117,6 +119,11 @@ function ModelProvidersPanel() {
     } else if (editing && editing.context_window) {
       body.context_window = 0; // field cleared → 0 tells the API to reset to NULL
     }
+    if (form.max_output_tokens && form.max_output_tokens > 0) {
+      body.max_output_tokens = form.max_output_tokens;
+    } else if (editing && editing.max_output_tokens) {
+      body.max_output_tokens = 0; // cleared → 0 resets to NULL
+    }
     try {
       if (editing) await updateModelProvider(editing.id, body);
       else await createModelProvider(body);
@@ -143,7 +150,14 @@ function ModelProvidersPanel() {
     setStatus(null);
     try {
       const r = await testModelProvider(p.id);
-      setStatus(`${p.name}: ${r.ok ? t("prov.testOk") : t("prov.testIncomplete")} — ${r.detail}`);
+      // A reachable endpoint whose key couldn't be verified (api_key_verified ===
+      // null) is NOT a clean pass — surface it as a caution, not "OK".
+      const label = !r.ok
+        ? t("prov.testIncomplete")
+        : r.api_key_verified == null
+          ? t("prov.testUnverified")
+          : t("prov.testOk");
+      setStatus(`${p.name}: ${label} — ${r.detail}`);
     } catch (e) {
       setStatus(String(e));
     }
@@ -195,6 +209,17 @@ function ModelProvidersPanel() {
               placeholder="1000000"
             />
           </Field>
+          <Field label={t("prov.fMaxOutput")} hint={t("prov.hintMaxOutput")}>
+            <TextInput
+              inputMode="numeric"
+              value={form.max_output_tokens != null ? String(form.max_output_tokens) : ""}
+              onChange={(e) => {
+                const v = e.target.value.replace(/[^0-9]/g, "");
+                setForm({ ...form, max_output_tokens: v ? parseInt(v, 10) : null });
+              }}
+              placeholder="4096"
+            />
+          </Field>
           <Field label={t("prov.fApiKey")} hint={editing && editing.has_api_key ? t("prov.hintKeep") : t("prov.hintNew")}>
             <TextInput
               type="password"

sidecar/app/agent_runtime/agent_service.py

@@ -117,5 +117,7 @@ def get_model_credentials(conn: sqlite3.Connection) -> dict[str, Any]:
         # Optional operator-declared context window (tokens); None → inferred from
         # the model name by model_budget. NOT a secret.
         "context_window": row["context_window"],
+        # Optional operator-declared max output tokens; None → inferred. NOT a secret.
+        "max_output_tokens": row["max_output_tokens"],
     }
 

sidecar/app/agent_runtime/model_budget.py

@@ -88,9 +88,15 @@ def context_window(model: str | None, explicit: int | None = None) -> int:
     return _DEFAULT_CONTEXT
 
 
-def max_output_tokens(model: str | None) -> int:
+def max_output_tokens(model: str | None, explicit_max: int | None = None) -> int:
     """The active model's provider-imposed MAX output tokens (best-effort). Used to
-    clamp the completion budget so we never send a max_tokens the provider rejects."""
+    clamp the completion budget so we never send a max_tokens the provider rejects.
+
+    ``explicit_max`` (an operator-declared cap from the model-provider config) wins
+    when positive — so a third-party/unknown model whose real ceiling is below the
+    substring-table default isn't handed a max_tokens its endpoint 400s on."""
+    if explicit_max and explicit_max > 0:
+        return explicit_max
     m = (model or "").lower()
     for sub, cap in _MAX_OUTPUT_TOKENS:
         if sub in m:
@@ -107,7 +113,8 @@ def tool_output_char_budget(model: str | None, explicit_window: int | None = Non
                max(TOOL_OUTPUT_CHARS_FLOOR, tokens * _CHARS_PER_TOKEN))
 
 
-def completion_token_budget(model: str | None, explicit_window: int | None = None) -> int:
+def completion_token_budget(model: str | None, explicit_window: int | None = None,
+                            explicit_max: int | None = None) -> int:
     """Completion (max_tokens) budget: raised only where the window clearly
     supports it, floored at the historical value, capped by the module ceiling AND
     by the model's real provider max-output so we never trigger a 400.
@@ -118,4 +125,4 @@ def completion_token_budget(model: str | None, explicit_window: int | None = Non
     rather than being handed the 16384 floor it would reject."""
     scaled = max(COMPLETION_TOKENS_FLOOR,
                  min(context_window(model, explicit_window) // 8, COMPLETION_TOKENS_CEILING))
-    return min(scaled, max_output_tokens(model))
+    return min(scaled, max_output_tokens(model, explicit_max))

sidecar/app/agent_runtime/session_action_tools.py

@@ -204,7 +204,7 @@ def review_bucket_config(provider_id: str, bucket: str) -> str:
 
     @function_tool
     def survey_account(provider_id: str, max_buckets: int = 0) -> str:
-        """Read-only account survey: enumerate visible buckets and detect evidence sources (access logs, inventory) across the account, persisting a profile the evidence-import flow can use. This is the COSTLY account tool — it makes live S3 calls across every visible bucket. Prefer the cheap persisted-profile readers when they can answer: query_account_profile for cross-bucket posture ("which buckets are public / unencrypted / no lifecycle?") and compare_to_last_survey for "what changed" both read the LAST survey with no new S3 calls. Run this only to establish a first profile or deliberately refresh a stale one. Returns a compact summary (counts + summary + public-exposure note), not raw key lists, for you to narrate. Does NOT surface a separate card. Use only when the user's request is about the account/buckets — NOT for local-file analysis or unrelated questions. The result includes has_prior_survey — when true, call compare_to_last_survey next to report what changed. max_buckets (optional, 1-500) raises the per-survey bucket cap for large accounts (default 100); the result's truncated flag tells you if buckets were left out. Args: provider_id; max_buckets?."""
+        """Read-only account survey: enumerate visible buckets and detect evidence sources (access logs, inventory) across the account, persisting a profile the evidence-import flow can use. This is the COSTLY account tool — it makes live S3 calls across every visible bucket. Prefer the cheap persisted-profile readers when they can answer: query_account_profile for cross-bucket posture ("which buckets are public / unencrypted / no lifecycle?") and compare_to_last_survey for "what changed" both read the LAST survey with no new S3 calls. Run this only to establish a first profile or deliberately refresh a stale one. Returns a compact summary (counts + summary + public-exposure note), not raw key lists, for you to narrate. Does NOT surface a separate card. Use only when the user's request is about the account/buckets — NOT for local-file analysis or unrelated questions. The result includes has_prior_survey — when true, call compare_to_last_survey next to report what changed. max_buckets (optional, 1-2000) raises the per-survey bucket cap for large accounts (default 100); the result's truncated flag tells you if buckets were left out. Args: provider_id; max_buckets?."""
         p = provider(provider_id)
         if p is None:
             return _err("Unknown provider_id. Use a configured provider.")
@@ -213,7 +213,7 @@ def survey_account(provider_id: str, max_buckets: int = 0) -> str:
         had_prior = bool(account_repo.recent_run_ids_for_provider(conn, provider_id, 1))
         start("survey_account", provider_name(provider_id))
         try:
-            mb = max(1, min(int(max_buckets), 500)) if max_buckets else None
+            mb = max(1, min(int(max_buckets), 2000)) if max_buckets else None
             body = RunCreate(run_type="account_discovery", provider_id=provider_id,
                              user_prompt=_DEFAULT_PROMPTS["account_discovery"],
                              session_id=session_id, max_buckets=mb)