Skip to content

refactor: group mm_embeddings per sequence at worker output.#1986

Open
wly-115 wants to merge 1 commit into
xLLM-AI:mainfrom
wly-115:refactor/mm_emebdding
Open

refactor: group mm_embeddings per sequence at worker output.#1986
wly-115 wants to merge 1 commit into
xLLM-AI:mainfrom
wly-115:refactor/mm_emebdding

Conversation

@wly-115

@wly-115 wly-115 commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Description

Move multimodal embedding grouping from the batch consumer into the worker
output. SampleOutput.mm_embeddings is now grouped by sequence
(vector<vector<Tensor>>, outer = sequence, inner = one tensor per encoded
item in that sequence: images for mm_embed, the full hidden for the
full-embedding task), and each sequence's embeddings are carried on its own
RawSampleOutput. Serialization (shared memory and RPC) now maps the outer
sequence to outputs[i] instead of re-slicing a batch-level tensor list in
the consumer.

Related Issues

Change Type

  • Bug fix
  • New feature
  • Performance improvement
  • Refactor
  • Documentation
  • Test
  • Build or CI

Pull Request Checklist

Thank you for contributing to xLLM. Before requesting review, please make sure the following items are complete.

PR Title and Commit Messages

  • The PR title and each commit message follow the xLLM commit format: <type>: <subject>.

Allowed types: feat, bugfix, docs, test, refactor, chore, style, revert, perf, model, build, release.
The subject should use clear English, start with a verb, include at least 4 words, and end with ..

Pre-commit Checks

  • I have installed pre-commit by running pip install pre-commit or an equivalent command.
  • I have installed the hooks with pre-commit install.
  • I have run pre-commit run --all-files and fixed any reported issues.

If you are unsure how to set up pre-commit, see the pre-commit documentation.

Self Review

  • I have self-reviewed the code according to .agents/skills/code-review/references/custom-code-style.md, especially code written or assisted by AI.
  • I have rebased this PR onto the latest main branch.

Build and Test Coverage

  • Tests have been added or updated as needed.
  • CUDA: python setup.py build test has passed on a CUDA machine.
  • NPU: python setup.py build test has passed on an NPU machine.
  • MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes

@wly-115
wly-115 marked this pull request as draft July 21, 2026 03:38
@wly-115
wly-115 force-pushed the refactor/mm_emebdding branch 2 times, most recently from dc6ad0e to b6fda38 Compare July 21, 2026 07:12
@wly-115
wly-115 marked this pull request as ready for review July 21, 2026 07:15
@wly-115
wly-115 force-pushed the refactor/mm_emebdding branch from b6fda38 to fccaef6 Compare July 21, 2026 09:12
Co-Authored-By: Claude <noreply@anthropic.com>
@wly-115
wly-115 force-pushed the refactor/mm_emebdding branch from fccaef6 to 98c315b Compare July 21, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants