perf: eliminate validate-to-draft bubbles with asynchronous device-side handoff.

[maojunx99]

Description

The NPU becomes idle between target-model validation/sampling and the next draft-model forward when MTP is enabled.
The previous execution path waited for validation results on the host, copied accepted-token state from device to host, rebuilt the next decode input on CPU, and copied the metadata back to device before launching the draft model. Stream
synchronization and scheduler serialization made this latency directly visible as a validate-to-draft bubble.

Changes

• Overlap next-iteration draft preparation with target-model validation.
• Keep accepted tokens, embeddings, positions, KV sequence lengths, and cache
  slot calculation device-resident across MTP iterations.
• Use stream events and same-stream FIFO ordering for dependencies instead of
  blocking the scheduling thread with unnecessary stream synchronization.
• Prelaunch the next first-draft forward after rejection sampling:
  • use fixed-shape repair/current inputs;
  • repair the previous-token KV entry when all draft tokens are accepted;
  • redirect the unused repair write to a scratch slot after rejection to avoid
    overwriting valid KV cache data.
• Add host-to-device transition priming so a new or changed batch first follows
  the canonical scheduler state before switching to the device-resident path.
• Correct the position and KV sequence-length bases used by the asynchronous
  handoff, including the accepted-prefix offset during the first transition.
• Preserve TP-rank consistency for accepted tokens and next-step decode state.
• Reuse a persistent pinned accepted-token buffer instead of allocating pinned
  host memory every iteration.
• Add the dedicated MtpPrepareNextDraft AscendC fused operator to replace the
  sequence of small Torch operators used for:
  • accepted-length calculation;
  • last/previous token and embedding selection;
  • position and KV-length updates;
  • cache-slot generation.
• Keep explicit fallback paths for unsupported layouts and configurations.
• Encapsulate prelaunch/fused-kernel eligibility checks to keep the main decode
  path readable.

Profiling Comparision (qwen3.5 2b tp=2 mtp=3)

before:

after:

Performance Statistics

Relevant PR

xLLM-AI/xllm-ops#18

Change Type

• Bug fix
• New feature
• Performance improvement
• Refactor
• Documentation
• Test
• Build or CI

Build and Test Coverage

• Tests have been added or updated as needed.
• CUDA: python setup.py build test has passed on a CUDA machine.
• NPU: python setup.py build test has passed on an NPU machine.
• MLU: python setup.py build test has passed on an MLU machine.

Reviewer Notes