What would you like?
Summary
Current multimodal intake does not preserve the ordering of text and attachments. Instead, it separates text from multimodal inputs and reconstructs the prompt layout later in model-specific schedule builders.
This works for the currently supported cases, but makes mixed-modality inputs (e.g. audio → image → audio) difficult to support correctly and requires model-specific reconstruction logic.
Current state
- Multimodal inputs are separated from text during intake.
- Schedule builders reconstruct the prompt by interleaving text and modality-specific walks.
- This assumes a limited prompt structure and requires separate implementations for Qwen3-Omni and BAGEL.
- Mixed-modality ordering is currently guarded against rather than supported.
Desired state
Introduce an order-preserving multimodal adapter that keeps the original sequence of text and multimodal items throughout preprocessing.
The adapter should:
- Preserve the original ordering of text and multimodal inputs.
- Use per-item placeholders so the prompt can be tokenized once.
- Produce placement information that schedule builders can consume directly.
- Be shared between Qwen3-Omni and BAGEL, leaving only model-specific walk generation in each implementation.
Benefits
- Correct handling of arbitrary multimodal ordering.
- Support for multiple attachments of the same modality without special cases.
- Eliminate model-specific prompt reconstruction logic.
- Remove the audio-only layout restriction and duplicate prompt-splitting/tokenization paths.
- Simplify future multimodal features by having a single ordering-preserving representation throughout the pipeline.
Acceptance criteria
- audio → image → audio prefills in that order,
- Text interleaved between two attachments is preserved
- N attachments of one modality work for N > 1, in order
- Token-for-token parity with today on single-image and single-audio requests
- Both models share the adapter (Owen omni + Bagel); no bare continue silently drops an item
- No serving regression on i2t/s2t/i2s/s2s at B=1..32, measured as a paired same-build A/B
- Out of scope: multi-turn role structure, encoder capture-bucket sizing, streaming attachments.
Alternatives or context
Was mentioned in #150
What would you like?
Summary
Current multimodal intake does not preserve the ordering of text and attachments. Instead, it separates text from multimodal inputs and reconstructs the prompt layout later in model-specific schedule builders.
This works for the currently supported cases, but makes mixed-modality inputs (e.g.
audio → image → audio) difficult to support correctly and requires model-specific reconstruction logic.Current state
Desired state
Introduce an order-preserving multimodal adapter that keeps the original sequence of text and multimodal items throughout preprocessing.
The adapter should:
Benefits
Acceptance criteria
Alternatives or context
Was mentioned in #150