fix+feat: v0.34.0 — engine correctness, lifecycle robustness, agent de-ossification, model config

[hxddh]

What changed

A three-angle audit (agent ossification, DuckDB engine math, state-machine lifecycle) plus the reviewed third-party findings. Bugs that misreported numbers, wedged state forever, or needlessly boxed the agent in.

🔴 Analysis-engine correctness

• E1 — integer columns stay Int64. A single unparsed log line (or missing inventory size) coerced the numeric column to float64 → DuckDB DOUBLE, so reports showed status codes as 404.0 and sizes above 2^53 lost precision. Fixed at ingest.
• E2 — error rates are "of requests", not "of lines". error_rate_4xx/5xx + the 206/404/403 shares divided by every ingested line (incl. null-status text-fallback rows), diluting them by the unparsed fraction. Now divide by parsed requests.
• E3 — inventory denominators reconcile. average_object_size = total_size / object_count; small_object_ratio is over known-size objects.
• E4 — age bucketing is UTC. The DuckDB connection pins TimeZone='UTC' so datediff vs current_timestamp is tz-independent (was ±1 bucket off-UTC).

🟠 Lifecycle robustness

• SM1 — a failed dataset-persist no longer wedges an import in importing forever (post-download persistence was outside any try/except and had no reconciler).
• SM2 — the blocking turn path resolves its turn_guard handle on a persist failure, like the streaming worker — otherwise every later turn in the session stalled 120s.
• SM3 — the loser of a concurrent import claim fails its orphan pending run.
• SM4/SM5 — a report_ready publish failure can't downgrade a completed run to failed, and the executor's failure branch won't overwrite a terminal state.

🟡 Model provider & budget (reviewed findings)

• P2a — the provider test no longer false-greens or false-reds. 404/405 on /models → "reachable, key unverified" caution (not a confident pass); empty base_url is valid (OpenAI default), not "incomplete".
• PB — the tool-output budget is a hard cap. A single output that would exceed the remaining budget is withheld with a valid JSON "too large — narrow it" envelope (was only counted after landing in context).

🟢 De-ossification (don't box the agent in)

• OSS1 — custom aggregation gained expressiveness without loosening the whitelist+bound-params floor: a 2nd group-by dimension (cross-tabs), day/weekday buckets, distinct_ips/distinct_keys/p99/min-max-bytes metrics, deterministic top-N tiebreaker — all fixed SQL fragments, zero raw-row exposure.
• MO — operator-declarable max_output_tokens clamps the completion budget (symmetric with context_window) so a lower-cap third-party endpoint doesn't 400. Migration 019 + full plumbing + a provider-form field.
• OSS2/OSS3/OSS4 — agent-memory recall scales with the model window (floored 50); the skill-load guard was raised (the elastic byte budget is the real bound); the survey ceiling 500 → 2000.

Frontend

• P2b — the 409 blocking-fallback captures its answer baseline BEFORE the turn (not from a GET after the 409), closing a race that hung the UI "running" for ~2.5 minutes.

What checks were run

• cd sidecar && pytest -q — 593 passed (+8 in tests/test_v0340_fixes.py; updated the budget/model/migration tests).
• ruff check app — clean.
• cd frontend && npm run typecheck && npm run build — clean.

Known gaps / deferred

• OSS5/OSS6 (elastic turn cap + context-echo caps) deferred as low-value — the elastic tool-output budget already governs the real concern.
• Frontend P2b/MO field verified by types + build (no headless UI harness).

Security invariants

No new writes, no shell, no destructive S3 op. The aggregate de-ossification only widened the fixed-fragment vocabulary — identifiers still resolve from the whitelist and all values stay bound params (no raw rows, no arbitrary SQL). Audit still records the actual SQL. Secrets handling unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M3YojFPzjkfMj6YYSwvzdx

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3fc2e56341

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Remove the committed dataset during persist cleanup

If any operation after datasets_repo.create() fails—for example, marking the import files or recording the audit event—the dataset row has already been committed because datasets.create() commits internally. This handler then deletes dest_dir but never removes that dataset row, leaving a failed run with an uploaded dataset whose stored_path no longer exists. Either make the persistence sequence transactional or explicitly delete the committed dataset before removing its file.

Useful? React with 👍 / 👎.