feat: unify tinfoil retry stack, add timeouts, surface real errors

[ital0]

GLM 5.2 via Tinfoil intermittently needed multiple retries; failures were invisible and the UI showed a generic error. This PR unifies the retry stack, bounds every hang vector, and surfaces the real failure cause.

What changed

Backend — Tinfoil proxy timeouts (backend/src/tinfoil/routes.ts)

• Time-to-first-byte timeout (30s, configurable): upstream headers not arriving in time → 504 with a stable tinfoil upstream timeout marker the frontend classifies.
• Stalled-stream watchdog (60s idle, configurable): mid-stream stalls abort the upstream and error the relayed stream (never truncate — bodies are opaque HPKE bytes, silent truncation would corrupt them). Implemented by extending the existing capStream (onIdle: 'terminate' | 'error', optional byte cap) instead of forking a second idle-relay implementation.
• Client aborts now propagate to the enclave fetch (AbortSignal.any), so cancelled sends stop upstream work.

Frontend — SecureClient lifecycle (src/ai/tinfoil-client.ts, extracted from fetch.ts)

• Attestation (ready()) is now bounded (15s): the SDK has no timeout/AbortSignal anywhere, and a chat send's abort signal does not cover the attestation await — a hung atc.tinfoil.sh previously hung the send forever. The bound applies to every call (not just the first) because an SDK-internal reset re-attests on the next ready().
• A failed or timed-out ready() now evicts the cached client, so the next send reconstructs instead of inheriting a wedged instance.
• Eviction predicate broadened: the SDK's concurrent-reset race escapes as a bare TypeError: … null (reading 'fetch') (no exported error class), which now also triggers eviction alongside KeyConfigMismatchError. All classification is by err.name — the tinfoil browser entry does not export the error classes, so instanceof is unusable.
• The extraction realizes the code-split the prewarm path always claimed: prewarm-system-model, model-catalog, and the voice engine now import the light tinfoil-client module instead of dragging the whole AI-provider pipeline into the startup chunk.

Unified per-turn retry budget (src/ai/retry-budget.ts)

• Previously three retry layers stacked multiplicatively with no shared cap: AI SDK internal (maxRetries default 2) × empty-response retry (×2) × outer chat auto-retry (×3).
• Now one per-turn budget (6 requests / 120s wall-clock) lives on the chat instance and is consumed at the agent-agnostic routing choke point (works for the built-in pipeline, the Pi harness, and ACP agents alike). SDK-internal retries are disabled (maxRetries: 0) so every request is counted. A typed consumer/probe split prevents layers from double-drawing.
• Budget resets on user send, manual retry, success, and abort — automation turns (sendAutomaticallyWhen) previously shared one budget until exhaustion.
• Retry backoff is now actually exponential (2s/4s/8s + jitter); it was linear despite the comment.

Error UX + telemetry

• Errors carry a kind through serialization (attestation / timeout / rate-limit / provider / network) and the chat error state shows cause-specific guidance instead of "Something went wrong" for every failure.
• Minimal telemetry: tinfoil_attestation (outcome/duration/client, deduped to one success per client instance), chat_auto_retry gains a reason, plus chat_retry_success and chat_retries_exhausted.

Fault-injection coverage (all deterministic, DI-based, no module mocks)

ready() rejection/hang/timeout, construction failure, key-config mismatch, the null-transport race, re-attestation hang after internal reset, TTFB timeout, mid-stream stall, client abort propagation, connection reset cleanup, 429/4xx/5xx/timeout/network classification, budget exhaustion across layers, backoff progression, and budget reset semantics (including the automation path).

Verification

• bun run check (type-check, lint, format, license): clean
• bun run test: full frontend suite green
• bun run test:backend: 938 pass, 0 fail
• Tinfoil SDK behavior claims verified against tinfoil@1.1.11 dist (ready() semantics, internal retry counts, missing timeout/signal support, error names, EHBP framing)

[github-actions]

Semgrep Security Scan

No security issues found.

[github-actions]

Preview environment destroyed 🧹

Stack preview-pr-1151 and its Cloudflare subdomain have been cleaned up.

[github-actions]

PR Metrics

Metric	Value
Lines changed (prod code)	+698 / -212
JS bundle size (gzipped)	🟢 585.4 KB → 585.8 KB (+408 B, +0.1%)
Test coverage	🟢 81.00% → 81.12% (+0.1%)
Performance (preview)	Preview not ready — Render deploy may have timed out
Accessibility	—
Best Practices	—
SEO	—

Updated Thu, 30 Jul 2026 19:49:48 GMT · run #2518

[github-actions]

🔭 thunder-deep-review (advisory)

Reviewed the diff — no issues to report. ✅ Never approves, never requests changes, never gates merge.
_{head: 09c8f3202d51 · mode: deep · deferred 0 item(s) already reported by other bots (best-effort dedup)}

[github-actions]

🔭 thunder-deep-review (advisory)

Complements the other bots — surfaces only what they did not flag. Never approves, never requests changes, never gates merge.
_{head: 44eb88983a0d · mode: deep · deferred 0 item(s) already reported by other bots (best-effort dedup)}

[github-actions]

🔭 thunder-deep-review (advisory)

Complements the other bots — surfaces only what they did not flag. Never approves, never requests changes, never gates merge.
_{head: bf2cd1e4eaff · mode: deep · deferred 0 item(s) already reported by other bots (best-effort dedup)}

[github-actions]

🔭 thunder-deep-review (advisory)

Complements the other bots — surfaces only what they did not flag. Never approves, never requests changes, never gates merge.
_{head: a7b3c96abc49 · mode: deep · deferred 0 item(s) already reported by other bots (best-effort dedup)}

[github-actions]

🔭 thunder-deep-review (advisory)

Reviewed the diff — no issues to report. ✅ Never approves, never requests changes, never gates merge.
_{head: f321107bc0a7 · mode: deep · deferred 0 item(s) already reported by other bots (best-effort dedup)}

[github-actions]

🔭 thunder-deep-review (advisory)

Complements the other bots — surfaces only what they did not flag. Never approves, never requests changes, never gates merge.
_{head: 2afaa16f2091 · mode: deep · deferred 0 item(s) already reported by other bots (best-effort dedup)}