micro_scheduler: targeted get_next_batch calls must not pop the TP-follow FIFO - #212
Draft
vasilevklart wants to merge 2 commits into
Draft
micro_scheduler: targeted get_next_batch calls must not pop the TP-follow FIFO#212vasilevklart wants to merge 2 commits into
vasilevklart wants to merge 2 commits into
Conversation
added 2 commits
August 8, 2026 19:24
…llow FIFO A ScheduleTPNode is a mandate from the TP group leader - rank 0 has already committed to the batch and blocks on the collective until every follower joins, and a popped message has no re-queue path on a follower. The targeted call path (the speculation fresh-rid merge in _try_speculate_next) is a discretionary consumer that may reject rids from the batch it is handed, so serving it from the TP FIFO could strand the popped message and hang the group at the next collective. This was the KNOWN GAP note in _try_speculate_next - unreachable while spec targets exclude parallel nodes, reachable the moment TP speculation lands. Refuse the combination at the source: targeted calls never touch the TP FIFO (the message stays queued for the unconditional scheduling path), replacing the per-target filters that made the overlap only partially safe. exclude_target keeps its filter - its caller (yield-away) submits unconditionally.
Same sys.path preamble as every other modular test. Without it, the venv's PEP-660 editable finder (a fallback meta-path hook pointing at the primary checkout) silently resolves mstar to a tree WITHOUT the change under test - the two targeted-call tests were failing against old code while the fence itself was correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_try_schedule_tp_followpops the leader'sScheduleTPNodefrom the FIFO before returning the batch. A TP-follow batch is a mandate: rank 0 has already committed to it and blocks on the collective inside the forward until every follower joins (per the existing note inget_next_batch), and a popped message has no re-queue path — followers cannot initiate scheduling for parallel nodes.The targeted call path (
target_node_name/target_graph_walk, used only by the speculation fresh-rid merge in_try_speculate_next) is a discretionary consumer: it may reject rids from the batch it is handed (the "in-flight rid wins" branch). Serving it from the TP FIFO can therefore strand a popped message — the follower never runs the batch, and the TP group hangs at the next collective. This is theKNOWN GAP (latent)note in_try_speculate_next: unreachable today only because spec targets exclude parallel nodes, reachable the moment speculative scheduling is enabled for lockstep-parallel nodes (the "disable … for now" in_can_speculate).Change
Refuse the combination at the source: targeted calls never touch the TP-follow FIFO; the message stays queued for the unconditional scheduling path. This replaces the per-target filters, which only made the overlap partially safe.
exclude_targetkeeps its filter — its caller (the yield-away path) submits what it receives unconditionally.No behavior change in any current serving path: with speculation disabled for parallel nodes, no targeted call can be handed a TP-follow batch today. This hardens the invariant ahead of that work.
Tests
test/modular/test_tp_follow_targeted_gate.py:exclude_targetskips the head but keeps it queuedThe two gate tests fail without the fix; the two behavior-preservation tests pass with and without it.