Skip to content

micro_scheduler: targeted get_next_batch calls must not pop the TP-follow FIFO - #212

Draft
vasilevklart wants to merge 2 commits into
mstar-project:mainfrom
vasilevklart:fix/tp-follow-targeted-pop
Draft

micro_scheduler: targeted get_next_batch calls must not pop the TP-follow FIFO#212
vasilevklart wants to merge 2 commits into
mstar-project:mainfrom
vasilevklart:fix/tp-follow-targeted-pop

Conversation

@vasilevklart

Copy link
Copy Markdown

Problem

_try_schedule_tp_follow pops the leader's ScheduleTPNode from the FIFO before returning the batch. A TP-follow batch is a mandate: rank 0 has already committed to it and blocks on the collective inside the forward until every follower joins (per the existing note in get_next_batch), and a popped message has no re-queue path — followers cannot initiate scheduling for parallel nodes.

The targeted call path (target_node_name / target_graph_walk, used only by the speculation fresh-rid merge in _try_speculate_next) is a discretionary consumer: it may reject rids from the batch it is handed (the "in-flight rid wins" branch). Serving it from the TP FIFO can therefore strand a popped message — the follower never runs the batch, and the TP group hangs at the next collective. This is the KNOWN GAP (latent) note in _try_speculate_next: unreachable today only because spec targets exclude parallel nodes, reachable the moment speculative scheduling is enabled for lockstep-parallel nodes (the "disable … for now" in _can_speculate).

Change

Refuse the combination at the source: targeted calls never touch the TP-follow FIFO; the message stays queued for the unconditional scheduling path. This replaces the per-target filters, which only made the overlap partially safe. exclude_target keeps its filter — its caller (the yield-away path) submits what it receives unconditionally.

No behavior change in any current serving path: with speculation disabled for parallel nodes, no targeted call can be handed a TP-follow batch today. This hardens the invariant ahead of that work.

Tests

test/modular/test_tp_follow_targeted_gate.py:

  • targeted calls — including an exactly-matching target — return nothing and leave the FIFO intact
  • untargeted calls still serve and drain the FIFO
  • exclude_target skips the head but keeps it queued
  • FIFO order survives refused targeted calls

The two gate tests fail without the fix; the two behavior-preservation tests pass with and without it.

Kirill Vasilev added 2 commits August 8, 2026 19:24
…llow FIFO

A ScheduleTPNode is a mandate from the TP group leader - rank 0 has
already committed to the batch and blocks on the collective until every
follower joins, and a popped message has no re-queue path on a follower.
The targeted call path (the speculation fresh-rid merge in
_try_speculate_next) is a discretionary consumer that may reject rids
from the batch it is handed, so serving it from the TP FIFO could strand
the popped message and hang the group at the next collective. This was
the KNOWN GAP note in _try_speculate_next - unreachable while spec
targets exclude parallel nodes, reachable the moment TP speculation
lands.

Refuse the combination at the source: targeted calls never touch the TP
FIFO (the message stays queued for the unconditional scheduling path),
replacing the per-target filters that made the overlap only partially
safe. exclude_target keeps its filter - its caller (yield-away) submits
unconditionally.
Same sys.path preamble as every other modular test. Without it, the
venv's PEP-660 editable finder (a fallback meta-path hook pointing at
the primary checkout) silently resolves mstar to a tree WITHOUT the
change under test - the two targeted-call tests were failing against
old code while the fence itself was correct.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant