fix(cli): retry transient mounted-drive reads (ENG-4049)

[SystemSculpt]

Problem

bl drive mounts performs an idempotent GET directly against a sandbox. During standby/resume or gateway placement, the gateway can return its typed 404 WORKLOAD_UNAVAILABLE response before any workload dispatch. The CLI immediately failed that read, creating a poor first-call experience and repeated Sentry reports even though a short wait often succeeds.

Tracked in ENG-4049. The platform contract/root follow-up is ENG-4071, related to ENG-2972.

Root cause

The CLI treated every sandbox 404 alike. It did not distinguish:

• a platform-owned, retryable, pre-dispatch availability response
• a normal not-found response
• a workload-owned response
• a transport failure or other status

Without that contract, a safe read could not recover without risking broad retries of mutations.

Fix

• add a retry function that has no method parameter and always issues GET
• use it only for listing mounted drives
• require all of: HTTP 404, code WORKLOAD_UNAVAILABLE, retryable true, and canonical platform provenance
• retry sequentially with 500ms exponential backoff capped at 30s and a 60s retry window
• honor caller cancellation and close every response before another attempt
• do not retry transport failures, malformed/legacy errors, ordinary 404s, other statuses/codes, or workload-owned lookalikes
• parse both typed gateway and legacy workload error bodies for local output
• keep ordinary handled HTTP errors expected under ENG-4048
• keep an exhausted exact platform-unavailable failure reportable, so this mitigation cannot hide the upstream defect

Verification

• make test
• go vet ./...
• go build ./...
• make lint: 0 issues
• focused race-enabled retry tests
• regression coverage proves:
  • two typed transient responses then success: 3 calls, max concurrency 1
  • exact backoff sequence and one user wait notice
  • retry budget exhaustion retains the typed actionable error and remains reportable
  • cancellation stops during backoff
  • every response body closes, including retries and final success
  • transport errors make one call
  • missing provenance, wrong code, retryable false, wrong status, and workload-owned lookalikes make one call
  • ordinary final HTTP errors remain expected under the Sentry taxonomy
• git diff --check

Scope

This is intentionally a narrow customer-path recovery for one idempotent read, not a general retry framework and not a substitute for the gateway contract fix. Mount, unmount, create, delete, and every other mutation remain single-attempt.

Depends on #352, which supplies the explicit telemetry classification boundary.

Linear: ENG-4049, ENG-4071

---

Note

Adds a retry mechanism for the idempotent GET used by bl drive mounts. Only retries on a precisely-identified platform-owned WORKLOAD_UNAVAILABLE 404 with exponential backoff (500ms initial, 30s max, 60s budget). Includes a comprehensive test suite covering success-after-retry, budget exhaustion, cancellation, transport errors, body closing, and rejection of non-retryable/untrusted error shapes.

^{Written by Mendral for commit e1a0716.}

[mendral-app]

🧪 Testing Guide

What this PR addresses

The bl drive mounts CLI command immediately failed with a 404 error when the sandbox was in a standby/resume state or the gateway hadn't yet dispatched the workload. This PR adds exponential-backoff retry logic specifically for the idempotent GET used by drive mounts, distinguishing the platform-owned WORKLOAD_UNAVAILABLE 404 (retryable) from workload-owned or non-retryable errors.

Steps to reproduce the original issue

1. Have a sandbox that is in standby or transitioning (cold start / gateway placement pending).
2. Run bl drive mounts --sandbox <sandbox-name>.
3. Before this PR: The CLI immediately exits with an error like failed to list mounted drives (HTTP 404) even though retrying a few seconds later would succeed.

What to verify (expected behavior)

1. Retryable 404 is retried: When a sandbox returns HTTP 404 with the gateway envelope {"error":{"code":"WORKLOAD_UNAVAILABLE","origin":"platform","retryable":true,...}}, the CLI should:
   • Print "Waiting for sandbox to become available..." on the first retry.
   • Retry with exponential backoff (starting at 500ms, doubling up to 30s, capped at 60s total budget).
   • Succeed transparently once the sandbox responds with 200.
2. Non-retryable errors are NOT retried: A 404 without the WORKLOAD_UNAVAILABLE code, without platform provenance (X-Blaxel-Source: platform header or origin: "platform" in the body), or with "retryable": false should fail immediately as before.
3. Budget exhaustion: If retries exhaust the 60s budget without success, the CLI should exit with a descriptive error including the WORKLOAD_UNAVAILABLE code and the gateway's action hint — and this error should NOT be marked as an "expected CLI error" (so it surfaces in Sentry).
4. Context cancellation: If the user hits Ctrl+C during a retry backoff, the CLI should exit promptly without waiting for the timer.
5. No regression for mutations: Only the idempotent GET /drives/mount path uses the retry logic; other drive operations (write, delete, etc.) are unchanged.
6. Unit tests pass: Run go test ./cli/ -run TestSandbox — all new tests in cli/drive_retry_test.go should pass, covering sequential retry, budget exhaustion, cancellation, transport errors, and response body closure.

Note

Posted by PR Testing Guide · Tag @mendral-app with feedback.

[mendral-app]

LGTM

Well-scoped, defensive retry for a single idempotent read path. The classification logic (isRetryableWorkloadUnavailable) is strict—requiring 404 status, correct code, retryable: true, and platform provenance. The retry loop correctly closes every response body, caps delay to remaining budget, and respects context cancellation. The test suite is thorough and the backoff arithmetic checks out. No correctness, security, or concurrency issues found.

_{Tag @mendral-app with feedback or questions. View session}

[SystemSculpt]

Parking this for now — not abandoned.

State: the fix is correct and targets a real, currently-active group. CLI-2X has 60 events in the last 7 days with exactly this signature: failed to list mounted drives (HTTP 404) … WORKLOAD_UNAVAILABLE.

Why it is parked:

• it is stacked on codex/eng-4048-cli-sentry-boundary (fix(cli): classify and sanitize Sentry failures (ENG-4048) #352), so no gate here has run against main yet
• the retry behaviour has not been verified end to end; that needs a sandbox in standby to produce a genuine pre-dispatch WORKLOAD_UNAVAILABLE

To resume: merge #352, let GitHub retarget this to main, re-run make test / make lint / go build, then verify against a standby sandbox before merging.

Related: ENG-4049, with the platform contract follow-up in ENG-4071.