fix(dq): port ducklake poison recovery to the query fleet + sustained-NotReady self-exit

[elffjs]

Summary

Fixes #21: the din-maintenance/inlined-data collision (INTERNAL Error: Attempted to access index 0 within vector of size 0 in ducklake v1.5.4's DuckLakeInlinedDataReader::TryInitializeScan) poisons query pods, and 1736873's two-strike recovery covered only the materializer's decode loop. A poisoned query pod parked Running-but-NotReady indefinitely — liveness probes the mon port and always passes, so nothing ever restarted it (observed on dsn-gcp: both pods 0/1 for ~3 days, x102 readiness failures, first mirrored read traffic 503ing).

This implements the issue's two independent in-repo fixes.

1. Driver-level two-strike poison recovery on the query pool (internal/service/duck/poison.go)

The query path has no retry loop to hook (unlike the materializer's Run loop), so recovery observes errors at the database/sql driver level: a connector wrapper installed in duck.NewService funnels every QueryContext/ExecContext outcome — GraphQL reads, fetch gRPC, and the /ready probe — into one recovery, with the same calibration as 1736873:

• An aborted catalog session (Current transaction is aborted) burns the idle pool so fresh sessions re-bootstrap (re-ATTACH).
• A concurrent pool has no "consecutive pass", so the second strike is judged by connection age: a session created after the recycle that is still aborted proves the poison is instance-level, and the process exits so the pod supervisor restarts it in seconds. Stragglers still in flight on pre-recycle sessions re-burn the pool but never escalate, and a healthy post-recycle query disarms the escalation so the next collision (din flushes every ~15m under read load) starts over at strike one.
• database has been invalidated escalates straight to restart, including when it surfaces at connect time.
• Recycles are counted in a new dq_duckdb_poison_recycles_total metric so collision frequency is alertable.
• The materializer's own duck service explicitly opts out (PoisonRecovery=false): its decode loop keeps the proven 1736873 recovery, and stacking the driver-level one would race it.

The wrapper forwards the exact optional-interface set *duckdb.Conn implements (pinned by a compile-time assertion) so database/sql behavior is unchanged for healthy traffic, including prepared statements.

2. Sustained NotReady exits the process (internal/app/ready.go)

A readiness failure sustained for 5 minutes with no intervening success terminates the process. It wraps the raw backend check inside loadTolerantReadiness's cache, so it judges real backing probes; 5m is well past the 60s cascade-grace window, and it backstops anything the poison classifier doesn't catch.

Not addressed here

• Issue fix 3 (upstream ducklake bump): the extension version follows the pinned duckdb-go (v2.10504.0 → DuckDB v1.5.4) via installext; no clearly matching upstream fix found, so reporting upstream with the issue's stack is still warranted.
• Issue fix 4 (shrinking din's flush collision surface) is a din-side change.

Testing

• New unit tests pin the error classifiers (including that the collision's raw INTERNAL error is deliberately not treated as pool poison — only its aftermath is), the age-based two-strike logic (straggler vs fresh-session strikes, disarm on post-recycle success, invalidated-instance escalation), wrapper observation on both query paths via a stub conn, and wrapper transparency for healthy traffic and prepared statements on a live in-memory DuckDB.
• New readiness test pins the sustained-failure exit and window reset on success.
• go build, go vet, and all runnable package tests pass. The tests/ and duck integration suites fail only on this sandbox's blocked extension downloads (HTTP 403 from extensions.duckdb.org) — verified identical on the pristine tree.

🤖 Generated with Claude Code

https://claude.ai/code/session_013AeGu5b99yoXAhVcBBxGUw

---

Generated by Claude Code