Pi/Minimax SDK errors cascade under concurrent load — needs throttling, better classification, and richer error surface

[Wirasm]

Summary

• What broke: When many Pi/Minimax sessions run concurrently — typical during multi-PR review batches and any workflow that fans out parallel AI nodes — sessions hit dag.node_sdk_error_result and abort. Heavy code-review prompts fail more often than light aspects (test-coverage, comment-quality, error-handling). Two distinct failure modes observed.
• When it started: Surfaced today (2026-05-04) running 11 maintainer-review-pr workflows in parallel (~100 Pi sessions in flight). Likely existed before — earlier sessions had partial failures we attributed to flake.
• Severity: major — review batches return mostly empty findings; users either re-run sequentially (slow, expensive) or merge without deep review.
• Related: Workflow runs zombie in 'running' state when DAG node hits SDK error mid-stream #1561 fixed the symptom (zombie workflow runs after these errors). This issue is about the cause — preventing the errors in the first place.

Two failure modes observed

Fast cascade (1–12s)

All aspects on a PR crash near-simultaneously, in single-digit seconds. Looks like rate-limit-and-give-up.

Example — PR #1548 review (today's batch):

nodeId=code-review     errorSubtype=error  durationMs=1869   stopReason=error
nodeId=test-coverage   errorSubtype=error  durationMs=2260   stopReason=error
nodeId=comment-quality errorSubtype=error  durationMs=2261   stopReason=error
nodeId=error-handling  errorSubtype=error  durationMs=2327   stopReason=error
nodeId=docs-impact     errorSubtype=error  durationMs=2327   stopReason=error

5 aspects, all error, all in <2.5s. PR #1529 same pattern.

Slow cascade (long-running session eventually errors)

A Pi session runs long, then errors. Looks like context overflow, hung tool call, or a stalled stream that the SDK eventually surfaces.

Example — repo-triage-minimax's closed-dedup-check:

durationMs=1885830  errorSubtype=error  stopReason=error

31 minutes before erroring. The node used the Claude-only agents: feature which Pi can't run; Pi accepted the prompt anyway, ran for 31m, then errored.

Evidence — today's 11-PR maintainer-review batch

PR	Aspects ran/scheduled	Pattern
#1523	4/4	clean (early in batch)
#1525	3/4	code-review crashed
#1554	1/4	3 aspects crashed
#1557	3/5	code-review, docs-impact crashed
#1533	1/4	3 aspects crashed
#1551	4/5	test-coverage crashed
#1553	1/5	4 aspects crashed
#1555	0/1	sole scheduled aspect crashed
#1529	0/5	all crashed
#1548	0/5	all crashed

Pattern: success rate degrades roughly with batch saturation. Code-review (heaviest prompt) crashes most often. Lighter aspects crash when the system is most loaded. The first PRs in the batch fared best; the later ones crashed almost completely.

By contrast, when a single Pi/Minimax workflow ran alone today (maintainer-standup-minimax, repo-triage-minimax, isolated maintainer-review-pr), they completed cleanly. The error pattern is concurrency-driven, not per-prompt-driven.

Steps to Reproduce

1. Launch ~10+ maintainer-review-pr workflows in parallel against open PRs:

   for pr in 1523 1525 1529 1533 1548 1551 1552 1553 1554 1555 1557; do
     archon workflow run maintainer-review-pr "review PR #$pr" &
   done

2. Watch logs for dag.node_sdk_error_result — most heavy-prompt nodes will fail in the back half of the batch.
3. The first ~3–4 reviews complete OK; the rest degrade as concurrent Pi-session count rises.

What we know about the SDK error

Today's logs surface very little upstream context:

{
  \"nodeId\": \"...\",
  \"errorSubtype\": \"error\",
  \"sessionId\": \"019df...\",
  \"stopReason\": \"error\",
  \"durationMs\": ...,
  \"msg\": \"dag.node_sdk_error_result\"
}

errorSubtype: \"error\" is the catch-all bucket. We don't know whether Pi sent a 429, a TLS reset, an upstream Anthropic 529, an internal Pi assertion, or a hung-stream timeout. Without the underlying error message, we can't classify these accurately as transient/fatal, which is why the existing transient-retry path doesn't trigger.

Relevant code: packages/workflows/src/dag-executor.ts:902-916 swallows the underlying SDK error message into a generic "SDK returned error" — only errorSubtype survives. The original msg.errors array (line 909) is logged but not propagated to the user-facing error message.

Suggested fixes (multi-tier)

1. Surface the underlying error (low effort, high diagnostic value)

In dag-executor.ts:914-916:

// Currently:
throw new Error(\`Node '\${node.id}' failed: SDK returned \${subtype}\${errorsDetail}\`);
// errorsDetail is built from msg.errors but Pi rarely populates it.

Pi's adapter (packages/providers/src/community/pi/event-bridge.ts) maps Pi events into Archon's MessageChunk stream. When Pi rejects with a status / network error / SDK assertion, that error string should reach the result chunk's errors[]. Currently it goes into a system chunk that we log and discard. Plumb it through.

2. Pi-side concurrency throttle (medium effort)

The Pi adapter is the right place for this — a semaphore that caps concurrent session.prompt() calls. Configurable via .archon/config.yaml:

assistants:
  pi:
    maxConcurrent: 4   # default that doesn't saturate Pi/Minimax

Honest fallback: when the limit's saturated, queue not error. Workflow nodes that would otherwise spawn 5 simultaneous Pi calls instead serialize through the semaphore.

This is a Pi-specific feature, not a workflow-engine feature — Claude SDK does its own throttling at the SDK layer; Pi doesn't.

3. Better transient-error classification (medium effort)

packages/workflows/src/executor-shared.ts (isTransientNodeError) currently checks for keywords like ECONNRESET, 429, etc. on the surfaced error string. Once #1 is in place, common Pi-side transients (rate-limit, upstream-timeout) will become detectable and trigger the existing retry-with-backoff path.

Today, every Pi SDK error gets the generic \"SDK returned error\" message which doesn't match any transient pattern → no retry → hard fail.

4. Per-PR review-pr workflow throttling (low effort, narrow fix)

The maintainer-review-pr workflow fans out 5 aspects in one DAG layer. For Pi-provider runs, the workflow could be rewritten to run aspects sequentially (or in pairs). This is a YAML-only change with no code impact — but it only fixes this one workflow, not the underlying issue.

Recommendation order

1. Fix Model stucked at response stream text #1 first — even if we did nothing else, surfacing the real error makes everything easier to diagnose. ~30 min of work.
2. Then doesn't know what o1-mini is, or how to route to openrouter.ai #3 — once errors are real, transient-retry starts working. Many of today's failures would have self-recovered.
3. Then updated code to use locally hosted llama LLM, nomic-embed-text model. #2 — the proper architectural fix, but more work.
4. Skip [FEATURE] How about bootstrapping the agent builder? #4 — narrow workaround that masks the real problem.

Adjacent bug (separate issue, mentioning here for cross-reference)

PR #1552's review during the same batch hit a different failure mode: condition_json_parse_failed on the gate's output. The gate produced JSON wrapped in markdown fences (\\\\json\\n{...}\\n\\\\\), and the condition-evaluator's \$gate.output.verdict == 'review' couldn't parse the field because \$gate.output came through as a string, not a parsed object. tryParseStructuredOutput (in event-bridge.ts) handles fence-stripping correctly when extracting the structured output from Pi, but the same logic isn't applied when downstream conditions reference the field. This warrants its own issue — not covered by any of the fixes above.

Environment

• Platform: CLI
• Database: SQLite
• Provider: pi / minimax/MiniMax-M2.7
• OS: macOS 25.3.0 (Darwin), Bun runtime

Impact

• Affected workflows: maintainer-review-pr, repo-triage-minimax, any DAG with parallel Pi nodes — primarily review and triage flows that fan out per-PR / per-issue work.
• Reproduction rate: Always when running 10+ concurrent Pi-review workflows. Intermittent at lower concurrency.
• Workaround: Run sequentially (1 at a time) or in batches of 2–3.
• Data loss risk: No (Workflow runs zombie in 'running' state when DAG node hits SDK error mid-stream #1561 ensures DB stays consistent). Just lost work — review aspects produce no findings, so the user merges blind or re-reviews manually.

Scope

• Package(s) likely involved: providers (community/pi), workflows
• Modules: providers:community/pi:event-bridge, providers:community/pi:provider, workflows:dag-executor (lines 902–916), workflows:executor-shared:isTransientNodeError

[Wirasm]

🔍 Investigation: Pi/Minimax SDK errors cascade under concurrent load

Type: BUG

Assessment

Metric	Value	Reasoning
Severity	HIGH	Major feature broken — parallel Pi workflows produce mostly empty review findings; sequential workaround defeats the purpose of parallel review batches
Complexity	MEDIUM	3 files changed + one new Semaphore helper; no schema/DB changes, moderate integration risk from the semaphore
Confidence	HIGH	Root cause confirmed by SDK type inspection — AssistantMessage.errorMessage exists and is never surfaced; fix path is clear and evidence-backed

---

Problem Statement

Three layered root causes, each building on the previous:

1. Error message not surfaced — buildResultChunk in event-bridge.ts ignores AssistantMessage.errorMessage when building the result chunk. The dag-executor receives errors: [] (empty), so the thrown error reads "SDK returned error" — which matches no transient retry pattern and leaves no diagnostic signal.

2. No concurrency cap on Pi — PiProvider.sendQuery() creates a fresh session per call with no throttling. Claude SDK throttles at SDK level; Pi does not. At ≈100 concurrent sessions, the backend rate-limits and sessions cascade-fail.

3. Missing Anthropic 529 / "overloaded" transient pattern — after Fix C1 lands, '429'/'rate limit'/'too many requests' will match. But Anthropic 529 ("overloaded") doesn't appear in TRANSIENT_PATTERNS.

---

Root Cause Analysis

C1 — error detail discarded in buildResultChunk

WHY: dag.node_sdk_error_result log has errorSubtype: "error", which matches no retry pattern
↓ BECAUSE: errorsDetail in dag-executor is empty since msg.errors is always [] for Pi
Evidence: packages/workflows/src/dag-executor.ts:909 — const errorsDetail = msg.errors?.length ? \ — ...` : ''; ↓ BECAUSE:buildResultChunknever populateserrors[] Evidence:packages/providers/src/community/pi/event-bridge.ts:144-152`:

...(isError ? { isError: true, errorSubtype: last.stopReason } : {}),
// ← `errors` field missing, even when last.errorMessage is populated

↓ ROOT CAUSE: AssistantMessage.errorMessage?: string (@mariozechner/pi-ai/dist/types.d.ts:130) is ignored in buildResultChunk

C2 — no Pi concurrency cap

WHY: Sessions fail in the "back half" of large batches (first PRs succeed, later ones crash)
↓ ROOT CAUSE: PiProvider.sendQuery() creates unlimited concurrent sessions with no semaphore
Evidence: packages/providers/src/community/pi/provider.ts:127-133

C3 — missing overloaded/529 pattern

After C1 lands, '429'/'rate limit' already in TRANSIENT_PATTERNS will match. Missing: '529' (Anthropic HTTP status) and 'overloaded' (error text).
Evidence: packages/workflows/src/executor-shared.ts:43-57

---

Implementation Plan

Step	File	Change
1	providers/src/community/pi/event-bridge.ts:144	Include errors: [last.errorMessage] in result chunk when isError && last.errorMessage
2	providers/src/community/pi/event-bridge.test.ts:181	Assert errors: ['auth'] in stopReason=error test; add "no errors when errorMessage absent" test
3	providers/src/types.ts	Add maxConcurrent?: number to PiProviderDefaults
4	providers/src/community/pi/config.ts	Parse maxConcurrent as positive integer in parsePiConfig
5	providers/src/community/pi/provider.ts	Add module-level Semaphore class + piSemaphore; wrap bridgeSession with acquire/release in try/finally
6	workflows/src/executor-shared.ts:43	Add '529' and 'overloaded' to TRANSIENT_PATTERNS
7	providers/src/community/pi/config.test.ts	Add maxConcurrent parse tests

📋 Step-by-step implementation details

Step 1: Surface Pi error in buildResultChunk

// packages/providers/src/community/pi/event-bridge.ts
// Change the isError spread inside buildResultChunk:

// Before:
...(isError ? { isError: true, errorSubtype: last.stopReason } : {}),

// After:
...(isError
  ? {
      isError: true,
      errorSubtype: last.stopReason,
      ...(last.errorMessage ? { errors: [last.errorMessage] } : {}),
    }
  : {}),

Step 5: Semaphore in provider.ts

// Add module-level class and variable before sendQuery:
class Semaphore {
  private available: number;
  private readonly waiters: (() => void)[] = [];
  constructor(count: number) { this.available = count; }
  acquire(): Promise<void> {
    if (this.available > 0) { this.available--; return Promise.resolve(); }
    return new Promise<void>(resolve => { this.waiters.push(resolve); });
  }
  release(): void {
    const next = this.waiters.shift();
    if (next) { next(); } else { this.available++; }
  }
}
let piSemaphore: Semaphore | undefined;

// In sendQuery, replace the try/catch around bridgeSession:
const maxConcurrent = piConfig.maxConcurrent;
if (maxConcurrent !== undefined && piSemaphore === undefined) {
  piSemaphore = new Semaphore(maxConcurrent);
  getLog().info({ maxConcurrent }, 'pi.semaphore_initialized');
}
if (piSemaphore !== undefined) await piSemaphore.acquire();
try {
  yield* bridgeSession(session, effectivePrompt, requestOptions?.abortSignal, outputFormat?.schema, uiBridge);
  getLog().info({ piProvider: parsed.provider }, 'pi.prompt_completed');
} catch (err) {
  getLog().error({ err, piProvider: parsed.provider }, 'pi.prompt_failed');
  throw err;
} finally {
  piSemaphore?.release();
}

Step 6: TRANSIENT_PATTERNS additions

// packages/workflows/src/executor-shared.ts
export const TRANSIENT_PATTERNS = [
  'timeout', 'econnrefused', 'econnreset', 'etimedout',
  'rate limit', 'too many requests', '429', '503', '502',
  '529',        // Anthropic HTTP 529 = service overloaded
  'overloaded', // Anthropic/Minimax overload message text
  'network error', 'socket hang up', 'exited with code', 'claude code crash',
];

Config example (user-facing)

assistants:
  pi:
    maxConcurrent: 4   # queue beyond 4 concurrent sessions instead of failing

---

Validation

bun test packages/providers/src/community/pi/event-bridge.test.ts
bun test packages/providers/src/community/pi/config.test.ts
bun run type-check && bun run lint && bun run validate

---

Explicitly out of scope

• The 31-min hang on closed-dedup-check (used Claude-only agents: with Pi) — capability warning already emitted; separate issue
• The condition_json_parse_failed on PR C1552 mentioned in the issue body — warrants its own issue

---

Next Step

To implement: /implement-issue 1569

---

Investigated by Claude • 2026-05-04

[Wirasm]

Issue Resolution Report

PR: #1572 (#1572)
Status: COMPLETE

---

Summary

Two root causes were addressed:

1. Error swallowing: AssistantMessage.errorMessage from the Pi SDK was never surfaced into errors[] on the result chunk, so the DAG executor's transient classifier never saw 429/529/overload errors — they were silently discarded, breaking the retry chain.
2. No concurrency cap: Without a semaphore, all parallel Pi nodes fired simultaneously. A single overloaded backend returned errors for every call at once, cascading failures across the entire DAG run.

---

Changes Made

File	Change
packages/providers/src/community/pi/event-bridge.ts	Surface errorMessage into errors[]; add warn log on tool result serialize failure
packages/providers/src/community/pi/event-bridge.test.ts	Assert errors[] populated; cover absent and empty-string errorMessage
packages/providers/src/types.ts	Add maxConcurrent?: number to PiProviderDefaults
packages/providers/src/community/pi/config.ts	Parse maxConcurrent (positive int only; drop invalid)
packages/providers/src/community/pi/config.test.ts	Tests for positive, invalid, and combined-fields parsing
packages/providers/src/community/pi/provider.ts	Module-level Semaphore + snapshot-before-await acquire/release; semaphore smoke tests
packages/workflows/src/executor-shared.ts	Add '529' and 'overloaded' to TRANSIENT_PATTERNS; add 6-test classifyError suite

---

Validation

✅ Type check | ✅ Lint (0 warnings) | ✅ Tests (all packages green)

---

Review & Self-Fix

A post-implementation review (4 agents: code-review, error-handling, test-coverage, comment-quality) surfaced 7 findings. All 7 were resolved or explicitly accepted:

• Fixed (6): semaphore race condition (snapshot-before-await), silent serializeToolResult failure (add warn log), inaccurate maxConcurrent JSDoc, missing semaphore init tests, missing classifyError tests, empty-string errorMessage edge case
• Accepted (1, LOW): 'overloaded' substring breadth — FATAL priority guard prevents false-positive retries; pattern is consistent with existing short-substring convention

---

Suggested Follow-up

"Pi semaphore: extract Semaphore class and add full unit test suite" (P2) — the module-level singleton makes comprehensive isolation testing difficult. Extracting to semaphore.ts would enable a standalone test suite covering concurrency, fairness, and error paths.

---

Resolved by Archon workflow 5f595e3db46cbaef81198a98f7a35a88