fix(queue): bound and isolate the notify-evaluate deliver fan-out

[bitfathers94]

fix(queue): bound and isolate the notify-evaluate deliver fan-out

The notify-deliver enqueue after notify-evaluate ran through an unbounded
Promise.all over every resolved delivery (which can outnumber the batch's
events one-to-many across channels), rejected the whole fan-out on the
first send failure, and left the failure completely unlogged. A retry of
that job recovers nothing since evaluateNotificationEvent only returns
newly-created rows, so a lost send stayed pending until the 10-minute
stranded-delivery sweep caught it.

Route the sends through mapWithConcurrency at a bounded, exported
NOTIFY_DELIVER_SEND_CONCURRENCY, same posture as the backfill-registered-repos
fan-out: every send is attempted exactly once regardless of an earlier
one's outcome, a failed send logs a structured notify_deliver_fanout_send_failed
line naming the delivery, and any failures still fail the job afterward by
throwing an aggregate error naming every lost delivery id.

Closes #10022

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-31 09:04:50 UTC}

2 files · 1 AI reviewer · no blockers · readiness 93/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This PR bounds the previously-unbounded Promise.all fan-out of notify-deliver enqueues to NOTIFY_DELIVER_SEND_CONCURRENCY via mapWithConcurrency, matching the existing backfill-registered-repos pattern in the same file. Each send is now attempted exactly once regardless of sibling failures, failures are logged with a structured notify_deliver_fanout_send_failed line naming the delivery id, and any failures still surface as a thrown aggregate error afterward so the job is marked failed. The change is well-tested: concurrency bound, partial-failure isolation-and-throw, full-success no-op, and zero-delivery edge case are all covered, and traces correctly to the real evaluateNotificationEvent created-only-return behavior described in the PR.

Nits — 4 non-blocking

• The console.error at src/queue/job-dispatch.ts:428 is intentional structured logging consistent with the sibling backfill-registered-repos pattern above it, not a stray debug leftover, but worth confirming it routes through the same log pipeline as other structured logs in this file.
• job-dispatch.ts is now ~439 lines per the size-smell signal; consider whether the notify-evaluate case's growing inline logic is a candidate for extraction as more fan-outs get this treatment, though this single addition doesn't yet justify a split.
• Consider whether NOTIFY_DELIVER_SEND_CONCURRENCY should be tuned independently from NOTIFY_EVALUATE_EVENT_CONCURRENCY based on observed queue send latency in production rather than defaulting to the same value of 5.
• The sendResults.length used in the error message denominator is equivalent to deliveries.length; using deliveries.length directly in the throw message would avoid an extra intermediate name to reason about (minor).

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#10022
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	⚠️ 20/25	Preflight is ready, but the PR body does not name the validation run.
Contributor workload	✅ 10/10	Author activity: 119 registered-repo PR(s), 82 merged, 3 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	bitfathers94; Gittensor profile; 119 PR(s), 3 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: moderate

Linked issue satisfaction

Addressed
The diff exports NOTIFY_DELIVER_SEND_CONCURRENCY and routes the notify-deliver enqueue through mapWithConcurrency with a catch-and-return {deliveryId, ok} mapper, logs a structured notify_deliver_fanout_send_failed line with deliveryId and reason on failure, and throws an aggregate error naming the failed/total counts and ids after all sends are attempted, matching the backfill-registered-repos pa

Review context

• Author: bitfathers94
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 119 PR(s), 3 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Add validation command/output.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.02%. Comparing base (32c5f27) to head (6fb70f3).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10140      +/-   ##
==========================================
+ Coverage   79.95%   80.02%   +0.06%     
==========================================
  Files         282      283       +1     
  Lines       58728    58936     +208     
  Branches     6930     6986      +56     
==========================================
+ Hits        46958    47161     +203     
- Misses      11480    11484       +4     
- Partials      290      291       +1     

Flag	Coverage Δ
backend	97.59% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/queue/job-dispatch.ts	97.59% <100.00%> (ø)

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.