Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs

Code release for "Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs" (Su, Yang, Li, Geiping; 2026).

The paper has three experimental sections, each with its own subfolder under this directory. The subfolders are self-contained and each can be set up and run independently.

Layout

sec5_efficiency/      Paper Section 5: Efficiency
                      Qwen3-1.7B / 4B with 2- or 3-stream interleaved packing.
                      Trains "solving-while-reading" and "auditing-while-solving".
                      Eval: GSM8K, MATH500, LogicNLI, SQuAD, ProofWriter, PubMedQA.

sec6_security/        Paper Section 6: Security
                      Qwen2.5-7B / Qwen3-4B with interleaved packing.
                      Trains on multi-stream-reconstructed Alpaca.
                      Eval: TensorTrust, Gandalf, Purple, RuLES, StruQ-ID/OOD,
                      IFEval (via ASIDE + lm-eval-harness).

sec7_monitorability/  Paper Section 7: Monitorability
                      Stream-8B (Qwen3-8B) and Stream-27B (Qwen3.5-27B) with
                      10 cognitive streams. Qwen3.5 uses per-stream
                      Gated-DeltaNet states.
                      Eval: AF eval-aware/sub-vocalization, monitor accuracy
                      (Meinke/Schoen 6-class), concern sub-vocalization.

Implementation Notes

Aspect	Sec 5 / Sec 6	Sec 7
Backbone	Qwen3 / Qwen2.5	Qwen3 / Qwen3.5 (incl. DeltaNet)
Model class	Qwen3ForMultiStream / Qwen2ForMultiStream	Qwen3ForCausalLM
Dataset format	.jsonl, single-row per timestep	.npz packed shards, 10-channel
Data construction	wait-$k$ (MetaMath, HotpotQA, Alpaca)	synthetic 10-stream tabular
Number of streams	2 (solving-while-reading) / 3 (auditing)	10 (User, Output, 8 thinking)
Entry point	train/train/train{_qwen3}.py	train/train/train_stream.py

Quick start

Pick the section you care about and follow its README:

• Sec 5 (Efficiency) — sec5_efficiency/README.md
• Sec 6 (Security) — sec6_security/README.md
• Sec 7 (Monitorability) — sec7_monitorability/README.md

Citation

@article{su_2026_multi-stream,
  title={Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs},
  author={Su, Guinan and Yang, Yanwu and Li, Xueyan and Geiping, Jonas},
  year={2026}
}