Linear API budget exhausted by per-issue poll loops; input-required backlog is a self-sustaining feedback loop

[vnovick]

Summary

On a project with a moderate backlog, the daemon issues ~90–110 Linear API requests per minute against Linear's ceiling of 2500/hour (41.7/min) — roughly 2.5× over budget. The hourly budget is exhausted ~25 minutes in, and for the remaining ~35 minutes every tracker operation fails, including state transitions (working_state / completion_state), dependency audits, and input-required replays.

Worse, the traffic is dominated by per-issue polling loops whose volume is proportional to the number of stuck issues, and one of those loops is required to unstick them — so once the budget is exhausted the system cannot recover on its own within the window.

Observed on v0.2.0 (6d1c67c), tracker linear, max_concurrent_agents: 10, polling.interval_ms: 60000.

Evidence

GET /api/v1/state during the incident:

"rateLimits": {
  "requestsLimit": 2500,
  "requestsRemaining": 0,
  "requestsReset": "2026-08-03T22:54:36+03:00",
  "complexityLimit": 3000000,
  "complexityRemaining": 2999817
}

complexityRemaining is essentially untouched — this is request count exhaustion from many tiny queries, not query weight.

Once the budget is gone, every request logs linear: graphql error response, which makes the true request rate exactly measurable:

59  T21:40      103 T21:45      111 T21:48
86  T21:41      109 T21:46      152 T21:49
81  T21:42      108 T21:47       91 T21:50
99  T21:43       74 T21:51
87  T21:44       43 T21:52

Attribution over that same 13-minute window (1,203 total requests):

requests	share	source
358	30%	automation: input-required replay detail fetch
337	28%	orchestrator: dependency audit refresh
185	15%	orchestrator: tracker-reply check
137	11%	orchestrator: pending input resume detail fetch
~186	15%	candidate polls, worker fetches, comments, state transitions

84% of all Linear traffic is four per-issue re-fetch loops. Actual work — moving an issue, posting a comment — is in the remaining 15%, and it is what starves.

Root causes

1. input-required replay re-fetches every entry every 15s. cmd/itervox/automations_input_required.go:135 — replayInputRequiredIssueDetail takes a cache map[string]*domain.Issue, but that cache is constructed per invocation, so it never survives a tick. With the automation watcher on a 15s ticker (cmd/itervox/automations.go:78), N input-required entries cost N × 4 requests/minute indefinitely. The backlog on this project went 5 → 19 entries during a single investigation, i.e. 20 → 76 req/min from this loop alone.

2. Dependency-audit refresh has a per-tick budget but no staleness TTL. internal/orchestrator/dependency_audit.go:250. The only skip condition is entry.LastAuditedAt.Equal(now) (line 268), so all 59 audit rows cycle through the 20-row budget (dependencyAuditRefreshPerTickBudget, line 25) forever, whether or not anything changed. Steady-state cost with nothing happening in the project.

3. checkTrackerReplies is uncapped. internal/orchestrator/event_loop.go:553 iterates every InputRequiredIssues entry and calls FetchIssueDetail per entry, with no per-tick budget like the dependency audit has. Each result additionally calls auditFetchedIssueDependenciesAndDispatch.

4. UpdateIssueState costs 2 requests and caches nothing. internal/tracker/linear/client.go:255. Step 1 resolves state name → UUID via issue → team → states; step 2 mutates. The team's workflow-state map is static, but it is re-resolved on every single transition. The completion transition retries up to 4× (internal/orchestrator/worker.go:930), so a failing move to completion_state burns up to 8 requests and still fails.

The feedback loop (why it can't recover)

internal/orchestrator/event_loop.go:628 — resuming a pending_input_resume entry requires a successful FetchIssueDetail:

detailed, err := o.tracker.FetchIssueDetail(ctx, entry.IssueID)
if err != nil {
    slog.Warn("orchestrator: pending input resume detail fetch failed", ...)
    continue
}

So:

1. Issues accumulate in input-required / pending_input_resume.
2. Loops 1–3 re-poll every one of them on every tick → budget exhausted.
3. The resume path needs one more Linear read → it 429s → continue.
4. The entry stays queued, so it keeps being polled → budget stays at zero.

Observed directly: input-responder entries sat in the automation queue with reason: pending_input_resume for 75+ minutes while automationQueue never drained. The backlog that causes the exhaustion is the same backlog that the exhaustion prevents draining.

Secondary effect: successful work gets parked

internal/orchestrator/worker.go:963 — when the completion_state transition fails all 4 attempts, the issue is added to userCancelledIDs, which the exit handler converts into a pause. So under sustained rate limiting, an issue whose agent run succeeded can end up paused with its work done and the tracker never updated. This had not yet fired on our deployment (0 occurrences) but the budget sat at 0 for 35 minutes of every hour, so it is reachable.

Related: issues paused this way are indistinguishable from human-cancelled ones, because PausedIdentifiers is map[identifier]issueID and records no reason.

Suggested solutions

Ordered by impact-to-effort.

A. Cache the Linear workflow-state map (internal/tracker/linear/client.go:255). Team-scoped name → UUID, effectively static; a process-lifetime map keyed by team ID with a long TTL halves the cost of every transition and removes the fetch state id failure mode entirely. Smallest patch, clearest win.

B. Persist the input-required detail cache across ticks (cmd/itervox/automations_input_required.go:135). Hoist the cache to the watcher with a short TTL (30–60s), or skip re-fetching an entry whose QuestionCommentID and QueuedAt are unchanged since the last successful fetch. Removes the largest single consumer.

C. Add a staleness TTL to the dependency-audit refresh (internal/orchestrator/dependency_audit.go:268). Skip rows whose LastAuditedAt is newer than some interval (e.g. 10 min) instead of only those audited at exactly now. Keeps the existing priority ordering and per-tick budget, but makes an idle project cost ~0 instead of 20 requests/tick.

D. Cap checkTrackerReplies with a per-tick budget (internal/orchestrator/event_loop.go:553), mirroring dependencyAuditRefreshPerTickBudget, with oldest-first ordering so no entry starves.

E. Global tracker rate limiter with write priority. Track requestsRemaining (already parsed into rateLimits) and, below a reserve threshold, shed polling reads while still admitting writes — state transitions, comments, and the FetchIssueDetail on the resume path. This is what breaks the feedback loop: reads should never be able to starve the operations that let the queue drain. Backing off polls proportionally to remaining budget would also stop the daemon from spending its whole window in the first 25 minutes.

F. Record a pause reason in PausedIdentifiers (currently map[identifier]issueID). Distinguishing rate_limited / retries_exhausted / user_cancelled / transition_failed would let a transition-failure pause be retried automatically instead of needing a human, and would make the worker.go:963 behaviour recoverable.

Happy to send a PR for A–D if the approach looks right.

[vnovick]

Refinement on suggestion B — the waste is sharper than "the cache is per-tick", and the fix is smaller.

In cmd/itervox/automations_input_required.go, the fetch at line 86 happens before the already-fired check at line 93:

issue := replayInputRequiredIssueDetail(ctx, tr, detailCache, entry)   // line 86 — Linear request
if issue == nil { ... }

for _, automation := range automations {
    if _, alreadyFired := issueState.firedAutomationIDs[automation.ID]; alreadyFired {
        continue                                                        // line 93-95
    }
    ...
}

firedAutomationIDs is carried across ticks for an entry whose blockedKey is unchanged (line 70). So for a stable blocked entry where every active automation has already fired, the loop is guaranteed to dispatch nothing — and still pays a FetchIssueDetail every 15s, forever. That is the steady state for any issue sitting in input-required waiting on a human, which is exactly the population that grows during an incident.

Minimal fix — hoist the guard above the fetch:

pending := false
for _, automation := range automations {
    if _, fired := issueState.firedAutomationIDs[automation.ID]; !fired {
        pending = true
        break
    }
}
if !pending {
    next.issues[identifier] = issueState
    continue        // nothing can dispatch — don't spend a tracker request
}

issue := replayInputRequiredIssueDetail(ctx, tr, detailCache, entry)

This drops the 30%-of-all-traffic replay loop to ~zero in steady state without touching the cache lifetime at all, and it changes no dispatch behaviour: any entry with an unfired automation still takes the same path.

Note also that filter.max_age_minutes does not help here — it is evaluated inside matchesReplayInputRequiredAutomation, which runs after the fetch, so stale entries are paid for and then discarded. If the guard above lands, it may be worth moving the age check ahead of the fetch too.