[Bugfix] Fix mamba align mode sync scheduling correctness - #2
Closed
tdoublep wants to merge 5869 commits into
Closed
[Bugfix] Fix mamba align mode sync scheduling correctness#2tdoublep wants to merge 5869 commits into
tdoublep wants to merge 5869 commits into
Conversation
…nt (vllm-project#36135) Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…llm-project#34616) Signed-off-by: Or Ozeri <oro@il.ibm.com>
…1164) Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: zixiao <shunli.dsl@alibaba-inc.com> Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com> Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
…35246) Signed-off-by: Sage Moore <sage@neuralmagic.com>
Signed-off-by: 1195343015 <1195343015@qq.com> Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
…r leading to encoder_cache_size is 0 (vllm-project#35994) Signed-off-by: Miao, Avery <avery.miao@intel.com>
Signed-off-by: Xinyu Chen <xinyu1.chen@intel.com>
…ct#36133) Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
…project#36146) Signed-off-by: Tomer Asida <57313761+tomeras91@users.noreply.github.com>
…vllm-project#35794) Signed-off-by: Xin Yang <xyangx@amazon.com>
…ect#35775) Signed-off-by: Russell Bryant <rbryant@redhat.com>
…llm-project#35810) Signed-off-by: zhxchen17 <zhxchen17@fb.com>
…vllm-project#36176) Signed-off-by: Nick Hill <nickhill123@gmail.com>
…m-project#35384) Signed-off-by: dorhuri123 <dor.huri1@live.biu.ac.il>
vllm-project#36088) Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
…ul (vllm-project#36147) Signed-off-by: Nikhil Gupta <nikhil.gupta2@arm.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
…llm-project#36177) Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…gs` (vllm-project#36158) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…ject#34754) Signed-off-by: Shiyan Deng <dsy842974287@meta.com> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
…36047) Signed-off-by: Shiyan Deng <dsy842974287@meta.com> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
…lm-project#35478) Signed-off-by: Shiyan Deng <dsy842974287@meta.com> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
…ix (vllm-project#36208) Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Xiang Shi <realkevin@tutanota.com>
…llm-project#36274) Signed-off-by: jennyyyyzhen <yzhen@hmc.edu> Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
…lm-project#33881) Signed-off-by: Or Ozeri <oro@il.ibm.com> Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
…roject#36129) Signed-off-by: baoloongmao <baoloongmao@tencent.com> Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
…ing (vllm-project#35461) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
…uning (vllm-project#36683) Signed-off-by: Yanan Cao <gmagogsfm@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…uring config key computation (vllm-project#36563) Signed-off-by: Yanan Cao <gmagogsfm@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…llm-project#36698) Signed-off-by: Yanan Cao <gmagogsfm@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
…tectures (vllm-project#36499) Signed-off-by: Matthias Gehre <matthias.gehre@amd.com>
…hput improvement (vllm-project#36710) Signed-off-by: yewentao256 <zhyanwentao@126.com>
vllm-project#36829) Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
vllm-project#36831) Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
…oject#35895) Signed-off-by: sfeng33 <4florafeng@gmail.com>
Signed-off-by: louie-tsai <louie.tsai@intel.com>
Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>
…lm-project#36824) Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
…lure (vllm-project#36761) Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
…llm-project#36677) Signed-off-by: Yanan Cao <gmagogsfm@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…index` (vllm-project#36144) Signed-off-by: Yan Ma <yan.ma@intel.com>
…project#36813) Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
…treaming (vllm-project#29947) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
…rop (vllm-project#36670) Signed-off-by: István Ketykó <istvan.ketyko@gmail.com>
…llm-project#34328) Signed-off-by: phaedonsun <phaedonsun@tencent.com> Co-authored-by: phaedonsun <phaedonsun@tencent.com>
…llm-project#36599) Signed-off-by: AuYang <459461160@qq.com>
…ect#36806) Co-authored-by: Claude <noreply@anthropic.com>
…rage The num_speculative_blocks subtraction in MambaManager.remove_skipped_blocks() was applied unconditionally, but it is only needed for async scheduling where num_computed_tokens can include unconfirmed draft tokens. With sync scheduling, num_computed_tokens is already accurate, and the subtraction causes delayed block freeing (wasting memory under pressure). Changes: - Add async_scheduling field to MambaSpec, set from scheduler config - Gate the subtraction in remove_skipped_blocks() on async_scheduling - Gate the fake_execute_model_fn correction in tests on async scheduling - Parameterize test_mamba_prefix_cache for both sync and async modes with separate expected StepAction sequences for each Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
num_speculative_blockssubtraction inMambaManager.remove_skipped_blocks()was applied unconditionally, but is only needed for async scheduling wherenum_computed_tokenscan include unconfirmed draft tokens. With sync scheduling,num_computed_tokensis already accurate and the subtraction delays block freeing, wasting memory under pressure.async_schedulingfield toMambaSpec(set from scheduler config) and gated the subtraction on it.test_mamba_prefix_cachefor both sync and async scheduling modes with separate expectedStepActionsequences, and gated thefake_execute_model_fntoken correction on async mode.Test plan
pytest -xsv tests/v1/e2e/test_mamba_prefix_cache.py::test_mamba_prefix_cache— both[async]and[sync]variants pass on H100Fixes a regression from vllm-project#33997.
🤖 Generated with Claude Code