fix: bound stuck generation recovery#563
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
📝 WalkthroughWalkthroughReworks ChangesStuck Recovery Batching
AGENTS.md Session Completion Update
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Job as recoverStuckGenerations
participant DB as generationRequests table
participant Task as processGenerationTask
Job->>DB: select stale request IDs (status, cutoff, limit batchSize)
alt no stale IDs found
Job-->>Job: return { recovered: 0 }
else stale IDs found
Job->>DB: update selected IDs to pending/updatedAt
loop for each recovered request
Job->>Task: trigger(requestId, concurrencyKey, idempotencyKey, idempotencyKeyTTL)
end
end
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Greptile SummaryThis PR limits each stuck-generation maintenance pass. The main changes are:
Confidence Score: 4/5The generation recovery path can leave reset requests without a dispatched processing task.
packages/jobs/src/trigger/recover-stuck-generations.ts Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Hourly maintenance] --> B[Select stale generation requests]
B --> C[Reset selected rows to pending]
C --> D[Trigger process-generation task]
D --> E[Generation processing]
D -. dispatch failure .-> F[Pending row without recovery retry]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[Hourly maintenance] --> B[Select stale generation requests]
B --> C[Reset selected rows to pending]
C --> D[Trigger process-generation task]
D --> E[Generation processing]
D -. dispatch failure .-> F[Pending row without recovery retry]
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 9 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/persistence-postgres/drizzle/0021_confused_doctor_strange.sql">
<violation number="1" location="packages/persistence-postgres/drizzle/0021_confused_doctor_strange.sql:1">
P1: This migration builds new indexes with plain `CREATE INDEX`, which can block inserts/updates/deletes on these hot tables while each index is being built. On production-sized `messages` / generation tables, that can turn a deploy into a visible write outage. Using an online index build strategy (for example `CREATE INDEX CONCURRENTLY` run outside a transaction, or a staged migration process) would avoid that lock impact.</violation>
</file>
<file name="packages/jobs/src/trigger/recover-stuck-generations.ts">
<violation number="1" location="packages/jobs/src/trigger/recover-stuck-generations.ts:17">
P2: With `DEFAULT_RECOVERY_BATCH_SIZE = 25` and an hourly maintenance schedule, a worker outage leaving hundreds of requests stuck in `running`/`cancelling` would take many hours to fully recover (25 per hour). Consider whether 25 is sufficient for the expected failure scenarios, or whether a larger default (or adaptive batch sizing based on backlog depth) would better serve recovery SLAs.</violation>
<violation number="2" location="packages/jobs/src/trigger/recover-stuck-generations.ts:40">
P1: If `processGenerationTask.trigger()` throws mid-loop (e.g., rate limiting or network error), the batch UPDATE has already committed all rows to `pending`, but the remaining rows in the loop never get dispatched. Since the recovery query only selects `running` and `cancelling` statuses, those orphaned `pending` rows won't be picked up by future recovery runs—they'll be stuck indefinitely unless something else independently processes pending requests.
Consider either: (a) wrapping individual `trigger()` calls in try/catch so a single failure doesn't block the rest of the batch, or (b) dispatching before resetting status (trigger with idempotency, then update), or (c) using `batchTrigger()` to atomically enqueue all rows.</violation>
</file>
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
Contributor
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Duplication | 5 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Risk
Summary by cubic
Bound stuck generation/message recovery to capped batches and re-queue recovered requests instead of processing inline. Added staleness-safe updates, oldest-first selection with new indexes, idempotent enqueue (concurrency key + TTL), and rollback of resets if enqueue fails.
Dependencies
viteto 7.3.5,undicito 7.28.0 (and@expo/cli’sundicito 6.27.0); bumped internal packages.Migration
@blah-chat/persistence-postgres.Written for commit 18a28ca. Summary will update on new commits.
Summary by CodeRabbit
New Features
Bug Fixes
Performance
Documentation