Skip to content

fix(bootstrap): abort hung Cloudflare terminal handshake - #18

Merged
steipete merged 3 commits into
openclaw:mainfrom
SebTardif:fix/cf-terminal-handshake-timeout
Aug 16, 2026
Merged

fix(bootstrap): abort hung Cloudflare terminal handshake#18
steipete merged 3 commits into
openclaw:mainfrom
SebTardif:fix/cf-terminal-handshake-timeout

Conversation

@SebTardif

Copy link
Copy Markdown
Contributor

What Problem This Solves

Cloudflare workspace bootstrap upgrades the Crabbox terminal with fetch() and no abort signal. After connect, diagnostics already time out at 15s. A handshake that accepts TCP and never returns HTTP 101 can hang the isolate forever. The AWS dialer already sets handshakeTimeout: 15_000.

Evidence

Live Node against a TCP server that accepts the connection and never writes an HTTP response, then against the production cloudflareTerminalDialer path with a hung fetch.

$ node --version && uname -srm
v26.7.0
Darwin 25.6.0 arm64

$ node --import tsx /tmp/oc-crabhelm-handshake-proof.mjs
hung_server=http://127.0.0.1:52667/upgrade
no_signal=still-pending
timeout_fetch name=TimeoutError elapsed_ms=82
runtime_diagnostics message=Crabbox terminal handshake timed out elapsed_ms=15001
proof_ok hung Cloudflare terminal handshake aborted

Patched call site on this branch:

$ rg -n "handshakeTimeoutMs|controller.signal|handshake timed out" worker/bootstrap.ts aws/terminal-dialer.ts
worker/bootstrap.ts:565:const handshakeTimeoutMs = 15_000;
worker/bootstrap.ts:575:  const timer = setTimeout(() => controller.abort(), handshakeTimeoutMs);
worker/bootstrap.ts:579:      signal: controller.signal,
worker/bootstrap.ts:586:    if (controller.signal.aborted) throw new Error("Crabbox terminal handshake timed out");
aws/terminal-dialer.ts:5:const handshakeTimeoutMs = 15_000;
aws/terminal-dialer.ts:19:      handshakeTimeout: handshakeTimeoutMs,

A hung upgrade now fails in 15s with Crabbox terminal handshake timed out. The timer is cleared after a successful 101 so an accepted socket is not torn down.

Real behavior proof

  • Behavior or issue addressed: Cloudflare Crabbox terminal upgrade fetch() now carries a 15s handshake abort. A hung attach URL finishes as Crabbox terminal handshake timed out instead of leaving the Worker fetch pending.
  • Real environment tested: macOS Darwin 25.6.0 arm64, Node v26.7.0, crabhelm checkout /tmp/oc-impl-crabhelm-handshake on fix/cf-terminal-handshake-timeout. Live node against a local hanging HTTP server (accept, no response body) and the production default terminal dialer.
  • Exact steps or command run after this patch: Started a node:http server that never writes a response. Compared fetch with no signal (still pending at 250ms) to AbortSignal.timeout(80). Then called CrabboxWorkspaceBootstrap.runtimeDiagnostics() with a hung fetch so the compiled 15s handshake budget is the one in worker/bootstrap.ts.
  • Evidence after fix: terminal output copied below.
hung_server=http://127.0.0.1:52667/upgrade
no_signal=still-pending
timeout_fetch name=TimeoutError elapsed_ms=82
runtime_diagnostics message=Crabbox terminal handshake timed out elapsed_ms=15001
proof_ok hung Cloudflare terminal handshake aborted
  • Observed result after fix: Unbounded fetch stayed pending. The 80ms helper aborted with TimeoutError at 82ms. Production runtime diagnostics recorded Crabbox terminal handshake timed out at 15001ms. The abort timer is cleared after a successful upgrade.
  • What was not tested: Live Cloudflare Worker against a real Crabbox attach URL. AWS ws handshakeTimeout (already set in aws/terminal-dialer.ts).

Summary

  • Default Cloudflare terminal dialer aborts a hung WebSocket upgrade after 15s
  • Timer is handshake-only; it is cleared after HTTP 101 so the accepted socket stays up
  • Same budget as AWS handshakeTimeout and the existing post-connect diagnostics timer
  • Unbounded upgrade fetch landed in openclaw/crabhelm#8 (94d9d64, 2026-07-07) and has been present for 39 days
  • Same hang class as openclaw/clawrouter#112 and openclaw/octopool#61

The Cloudflare Crabbox terminal upgrade fetch had no AbortSignal.
Post-connect diagnostics already time out at 15s, but a handshake
that never returns HTTP 101 could hang the isolate. AWS already
uses handshakeTimeout 15_000. Clear the timer after a successful
upgrade so an accepted socket is not torn down.

Signed-off-by: Sebastien Tardif <sebtardif@ncf.ca>
@clawsweeper

clawsweeper Bot commented Aug 15, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Aug 15, 2026
@clawsweeper

clawsweeper Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed August 15, 2026, 10:42 PM ET / August 16, 2026, 02:42 UTC.

ClawSweeper review

What this changes

The PR adds a 15-second abort to the Cloudflare terminal WebSocket upgrade request, adds a focused timeout test, and adds an unreleased changelog entry.

Regression provenance

Possible regression — probable (reproduction; reviewed change). No predecessor PR is attributed.

Merge readiness

Blocked until stronger real behavior proof is added - 4 items remain

Keep this PR open: current main still has an unbounded Cloudflare terminal upgrade, and the patch is a sound bounded repair. Before merge, remove the release-owned changelog edit and provide redacted Cloudflare Worker or staging proof of the actual upgraded-fetch path.

Priority: P2
Reviewed head: afd56d49ccd326a9aef34691c5c32f5fb1a51ba5

Review scores

Measure Result What it means
Overall readiness 🦪 silver shellfish (2/6) The patch and focused test are good, but the required after-fix evidence does not yet cover the Cloudflare Worker behavior being changed.
Proof confidence 🦪 silver shellfish (2/6) Needs stronger real behavior proof before merge: The copied terminal transcript demonstrates local Node behavior and a mocked production call path, but not Cloudflare Workers’ upgraded-fetch behavior; add redacted deployed or staging diagnostics and update the PR body for re-review.
Patch quality 🐚 platinum hermit (4/6) 1 actionable review finding remain.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: The copied terminal transcript demonstrates local Node behavior and a mocked production call path, but not Cloudflare Workers’ upgraded-fetch behavior; add redacted deployed or staging diagnostics and update the PR body for re-review.
Evidence reviewed 6 items Current-main defect: The current default terminal dialer awaits the upgraded fetch without an AbortSignal, so a terminal endpoint that never completes the HTTP upgrade has no handshake bound.
Proposed repair: The branch creates a handshake-scoped AbortController, passes its signal to fetch, maps an abort to a clear error, and clears the timer in finally.
Focused regression coverage: The added test invokes the default dialer through runtime diagnostics, supplies a hanging fetch, advances 15 seconds, and asserts that the passed signal is aborted and the diagnostic rejects with the timeout error.
Findings 1 actionable finding [P3] Remove the release-owned changelog entry
Security None None.

How this fits together

Crabhelm’s Cloudflare control plane opens a bearer-authenticated outbound WebSocket to a Crabbox workspace terminal for runtime diagnostics. The dialer returns an accepted socket to bounded diagnostic collection, or should return a bounded error when the terminal handshake does not complete.

flowchart LR
  A[Operator diagnostics request] --> B[Cloudflare control plane]
  B --> C[Terminal dialer]
  C --> D[Crabbox upgrade endpoint]
  C --> E{Handshake within 15 seconds?}
  E -->|Yes| F[Accepted terminal socket]
  E -->|No| G[Bounded timeout error]
  F --> H[Redacted diagnostics result]
Loading

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: The copied terminal transcript demonstrates local Node behavior and a mocked production call path, but not Cloudflare Workers’ upgraded-fetch behavior; add redacted deployed or staging diagnostics and update the PR body for re-review.
  • Remove the release-owned changelog entry (P3) - CHANGELOG.md is release-owned, so normal PRs should not edit it. Keep the user-facing release context in the PR body or commit message and leave release-note selection to the release process.
  • Resolve merge risk (P1) - Cloudflare Workers’ upgraded-fetch abort behavior is not shown by the copied Node transcript or mocked fetch test, so a platform-specific handshake stall could remain undetected.
  • Complete next step (P2) - This external PR needs contributor-provided Cloudflare Worker proof before merge; the remaining code-policy cleanup is a small manual change.

Findings

  • [P3] Remove the release-owned changelog entry — CHANGELOG.md:5
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch surface production +19/-7, tests +39, release-owned changelog +1 The implementation is small and has focused test coverage; only platform-real behavior proof and release-note ownership remain.

Merge-risk options

Maintainer options:

  1. Prove the Cloudflare runtime path (recommended)
    Add redacted deployed or staging Cloudflare Worker evidence for a hung upgraded fetch and a successful accepted socket before merging.

Technical review

Best possible solution:

Keep the scoped timeout implementation and regression test, remove the release-owned changelog line, and land only after a redacted Worker or staging run proves both the 15-second failure path and an unaffected successful upgrade.

Do we have a high-confidence way to reproduce the issue?

No high-confidence Cloudflare Worker reproduction is supplied. Current source clearly shows the unbounded fetch and the branch test establishes the intended abort path, but the supplied live experiment is Node rather than the changed Worker runtime.

Is this the best way to solve the issue?

Yes for the code path: a handshake-scoped AbortController with finally cleanup is the narrowest repair and preserves accepted sockets. The release-owned changelog edit should be removed, and Worker-runtime proof is still needed.

Full review comments:

  • [P3] Remove the release-owned changelog entry — CHANGELOG.md:5
    CHANGELOG.md is release-owned, so normal PRs should not edit it. Keep the user-facing release context in the PR body or commit message and leave release-note selection to the release process.
    Confidence: 0.99

Overall correctness: patch is correct
Overall confidence: 0.9

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 71b80d86ca0b.

Labels

Label justifications:

  • P2: A hung terminal handshake can make control-plane inspection unavailable, but the effect is limited to the Crabbox terminal path.
  • merge-risk: 🚨 availability: This changes whether a Cloudflare Worker diagnostic request remains pending or fails after a fixed handshake budget.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The copied terminal transcript demonstrates local Node behavior and a mocked production call path, but not Cloudflare Workers’ upgraded-fetch behavior; add redacted deployed or staging diagnostics and update the PR body for re-review.

Evidence

What I checked:

  • Current-main defect: The current default terminal dialer awaits the upgraded fetch without an AbortSignal, so a terminal endpoint that never completes the HTTP upgrade has no handshake bound. (worker/bootstrap.ts:572, 71b80d86ca0b)
  • Proposed repair: The branch creates a handshake-scoped AbortController, passes its signal to fetch, maps an abort to a clear error, and clears the timer in finally. (worker/bootstrap.ts:565, afd56d49ccd3)
  • Focused regression coverage: The added test invokes the default dialer through runtime diagnostics, supplies a hanging fetch, advances 15 seconds, and asserts that the passed signal is aborted and the diagnostic rejects with the timeout error. (tests/worker-bootstrap.test.ts:511, afd56d49ccd3)
  • Product boundary: The architecture documents the deployment-specific Crabbox terminal as a bearer-authenticated server-to-server endpoint reached through a backend-specific outbound HTTPS WebSocket dialer. (docs/architecture.md:101, 71b80d86ca0b)
  • Feature provenance: The relevant current-main file history includes the merged AWS/control-plane introduction; provided PR context identifies that merged change as the origin of the unbounded Cloudflare terminal fetch. (worker/bootstrap.ts:572, 94d9d64fa0cb)
  • Review continuity and proof gap: Since the previous reviewed head, the only changed reviewed file is CHANGELOG.md; the PR body still explicitly says a live Cloudflare Worker against a real attach URL was not tested. (CHANGELOG.md:5, afd56d49ccd3)

Likely related people:

  • steipete: The merged control-plane work that introduced this dialer and the subsequent current-main bootstrap work are attributed to Peter Steinberger; the GitHub context identifies the corresponding contributor as steipete. (role: feature owner and recent area contributor; confidence: high; commits: 94d9d64fa0cb, 38d0dcc2792c, fcec701dc4cf; files: worker/bootstrap.ts, tests/worker-bootstrap.test.ts, docs/architecture.md)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Remove the release-owned CHANGELOG.md line.
  • Add redacted deployed or staging Worker output showing a hung handshake fails near 15 seconds.
  • Include a successful upgraded terminal connection in the same proof to show accepted sockets remain usable.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-08-15T23:54:43.986Z sha c3a057f :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-16T02:25:31.023Z sha fcec701 :: needs real behavior proof before merge. :: none

@steipete
steipete merged commit 904a530 into openclaw:main Aug 16, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. P2 Normal priority bug or improvement with limited blast radius. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants