fix: evict stale pooled connection before each control-channel dial

[mikhailm-coder]

Problem

During the mikart mesh-offline incident (Jul 1-2), agents entered a permanent retry loop logging Connection FAILED: No HTTP response (… tls=down, elapsedMs=0 …) every 4-6 minutes for 12+ hours, while /ws/nats connections from the same machines to the same IP/LB kept working. Evidence that the failure is local to the meshagent process:

• elapsedMs=0: hundreds of consecutive attempts failed in the same millisecond as the dial — impossible for any network exchange.
• The gateway's first request filter (logs every arriving upgrade) recorded almost none of the thousands of client-side attempts — the dials never left the machine.
• A TCP/TLS failure cannot be selective by URL path; NATS WS to the same address worked throughout.
• Every observed recovery coincided exactly with a meshagent process start (attempt=…-1); the one machine that never restarted (attempt counter 1→2533) never recovered.

Mechanism

ILibWebClient_PipelineRequestEx2 reuses a kept-alive connection from the manager's DataTable when an entry exists for the target address (microstack/ILibWebClient.c "Previous connection exists!" path). If that pooled socket was abortively closed by the gateway (SO_LINGER=0 RST) or died during a network blip, the queued upgrade fails immediately with no HTTP bytes — MeshServer_ControlChannel_ConnectSink never fires (hence tls=down), and each retry re-queues onto the same dead object.

Fix

Call ILibWebClient_DeleteRequests(agent->httpClientManager, &meshServer) immediately before pipelining the upgrade in MeshServer_ConnectEx, evicting any pooled connection for the target address so every dial performs a fresh TCP+TLS connect. Connection reuse buys nothing here: the control channel is one long-lived WebSocket, dialed at most once per hour when healthy.

Safety

• The single-flight guard (fix: single-flight guard in MeshServer_ConnectEx (prevent duplicate agent connections) #58) guarantees serverConnectionState == 0 && controlChannel == NULL && controlChannelRequest == NULL at this point, so there are no pending requests to abort — the call can only disconnect an idle stale socket.
• agent->httpClientManager has exactly one pipeline call site (the control-channel dial), so no other subsystem shares these pooled connections.
• ILibAsyncSocket_Disconnect synchronously invalidates the socket (internalSocket = ~0), so the subsequent PipelineRequest takes the reconnect path (SOCK == NULL || IsFree) rather than racing the teardown.
• Works for the proxy path too: the pool key is always meshServer (the address passed to PipelineRequest); proxy config is applied to the request token afterwards.

Testing

• CI PR build compiles all targets.
• Suggested manual repro: establish control channel, RST the connection mid-session (or drop DNS briefly), confirm the next dial performs a full TCP+TLS handshake (tls=up or success) instead of the instant elapsedMs=0 failure loop.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved connection reliability by ensuring new connection attempts start with a fresh network session instead of reusing stale cached requests.
  • Reduced failures caused by dead keep-alive connections when reconnecting to a server.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2af4b131-7665-43ff-9317-ab732b55ef5f

📥 Commits

Reviewing files that changed from the base of the PR and between 0da37a6 and 2adfbd5.

📒 Files selected for processing (1)

• meshcore/agentcore.c

---

📝 Walkthrough

Walkthrough

In MeshServer_ConnectEx, a call to ILibWebClient_DeleteRequests is added to remove stale or pooled HTTP requests for the resolved server address before the websocket request pipeline is created, ensuring a fresh connection attempt.

Changes

Stale Request Eviction

Layer / File(s)	Summary
Evict pooled requests before reconnect meshcore/agentcore.c	MeshServer_ConnectEx now calls ILibWebClient_DeleteRequests for the target meshServer address before building the websocket request pipeline.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

• flamingo-stack/meshagent#56: Both PRs modify MeshServer_ConnectEx's connection setup in the same request-creation path.
• flamingo-stack/meshagent#58: Both PRs modify MeshServer_ConnectEx to handle reuse of in-flight/pooled connection state.

Suggested reviewers: denys-gif

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: evicting stale pooled connections before control-channel dials.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/control-channel-fresh-connection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}