[IBKR] Dead Gateway connection reports health=healthy; orders accepted but never transmitted until manual reconnect (no keepalive)

[rudyll]

Summary

IBKR connection health is derived purely from query failure-counting (_consecutiveFailures), with no active keepalive/heartbeat. When the socket to IB Gateway dies silently (Gateway daily restart, network blip, half-open TCP), account.health stays "healthy" (false positive). Orders placed in this window are accepted at the UTA/wallet layer (get an orderId, status submitted) but never reach IBKR — they only transmit after a manual reconnect, at which point queued orders fill immediately.

Impact

• health: healthy while the connection is actually dead → callers place orders that silently don't execute.
• Account/position queries return stale cached data during the dead-but-"healthy" window.
• No auto-recovery; requires a manual reconnect.

Root cause (v0.40.0-beta.2)

1. No active heartbeat — requestCurrentTime() (services/uta/src/domain/trading/brokers/ibkr/request-bridge.ts:216) is on-demand only (e.g. getMarketClock), never on a timer. Nothing probes the idle socket.
2. connectionClosed() only rejects in-flight requests (request-bridge.ts:377 → rejectAll); it does not mark the account offline or trigger a reconnect.
3. Connectivity errors 502/504/1100 are effectively a no-op (request-bridge.ts:394-398 — comment "These will be followed by connectionClosed() which rejects all", then returns).
4. Health = passive failure count — UnifiedTradingAccount derives health from _consecutiveFailures (incremented only on a failed query, reset on success). A socket dying while idle never trips it → stays healthy. Order submits on the dead socket don't reliably trip it either.

Net: a silently-dropped connection is invisible to the health model until some query happens to fail — and order submits don't reliably trip it.

Repro

1. Connect an IBKR paper account via IB Gateway.
2. Let the OpenAlice↔Gateway socket drop silently while idle (e.g. Gateway daily auto-restart, or a brief network drop).
3. GET /api/trading/uta → account still health: healthy.
4. POST .../wallet/place-order → returns orderId, status submitted; it does not fill.
5. POST /uta/:id/reconnect → the order fills immediately.

Observed repeatedly on 0.40.0-beta.2 with an IB Gateway paper account: orders sat submitted (no fill) until we manually reconnected, then filled instantly.

Suggested fix

• Add an active heartbeat (periodic reqCurrentTime with timeout) that marks the account offline on failure.
• In connectionClosed() / on error 1100, mark the account offline and trigger (or schedule) a reconnect — not only reject in-flight requests.
• Treat error 1102 (connectivity restored) as a recovery signal.

[luokerenx4]

Implemented in #335 (merged) — needs verification on a real Gateway setup, and yours is the canonical repro environment.

Root cause confirmed, plus one mechanism beyond your analysis: the IBKR account surface is cache-backed, so routine health queries never touch the socket at all — a dead-but-idle connection is invisible to failure counting even in principle. That's why health: healthy survived a dead socket indefinitely.

What landed:

• Active heartbeat: reqCurrentTime every 45s with a 5s timeout; a failed beat marks the connection dead.
• connectionClosed() marks dead immediately (previously it only rejected in-flight requests).
• While dead, everything refuses loudly: cache reads (getAccount/getPositions) AND all order paths (place/modify/cancel) return "TWS/Gateway connection lost — reconnect pending… orders will NOT transmit" instead of accepting orders into the void.
• init() force-tears-down a half-open socket — isConnected() lies on half-open TCP, which previously made the recovery loop's re-init a silent no-op.

Not yet done from your list: treating error 1102 as an explicit recovery trigger (recovery currently rides the existing re-init loop, which the dead flag now actually un-blocks). Specs cover the gate wiring; what we could NOT reproduce locally is the silent half-open drop itself (we run TWS, not Gateway with daily restarts). If you can run your repro — Gateway daily auto-restart while idle — against master and confirm (1) health leaves healthy, (2) order placement refuses during the window, (3) it recovers without manual reconnect, we'll close this.