[RFC] Deep-dive OSDI 2026 Cloud LLM Inference Papers

[hsliuustc0106]

Summary

Request for interns to deep-dive the cloud LLM inference papers from OSDI 2026 (July 13–15, Seattle). 134 papers accepted, ~14 are directly relevant to our inference serving work. Each deep dive should produce: a paper summary, key insights relevant to InferMatrix, and a 5-minute lightning-talk slide deck.

Context

OSDI 2026 had 669 submissions (2× previous years), 20% acceptance rate. This is the first multi-track OSDI. Industry involvement from Alibaba, ByteDance, NVIDIA, Google, Microsoft, Tencent is heavy — many of these papers describe production systems at scale.

Papers to Deep Dive

KV Cache & Long Context

• Strata: Hierarchical Context Caching for Long Context Language Model Serving — Stanford, NVIDIA
• ECHO: Efficient KV Cache Offloading with Lossless Prefetching for Serving Native Sparse Attention LLMs — SJTU, Huawei
• No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs — UVA

Scheduling & Multi-Model Serving

• Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling — SJTU, Alibaba
• Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning — UCLA, UCB, CMU, ByteDance, Alibaba, LMSYS

Resource-Efficient & Heterogeneous Serving

• Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration — Sun Yat-sen
• Revisiting Pipeline Parallelism for LLM Serving — Korea Univ
• OpenTela: Unifying Decentralized Computing Resources for Heterogeneous LLM Serving ⚙️ — ETH Zurich, Cambridge, EPFL, MIT, HKUST
• Kairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing — Sun Yat-sen, EPFL, Peng Cheng Lab
• ADAngel: Accelerating Arbitrary-Precision Quantized LLMs with Adaptive Computing Mapping — SJTU

MoE Inference

• Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU–GPU Hybrid Design — Tsinghua, Xingyun IC
• BatchGen: An Architecture for Scalable and Efficient Batch Inference — Edinburgh, Tencent

GPU Networking

• UCCL-Tran: An Extensible Software Transport Layer for GPU Networking — UCB, Tsinghua, Harvard, IBM

Observability

• StriaTrace: Efficient Tracing and Diagnosis for Online LLM Inference ⚙️ — SJTU, Alibaba

⚙️ = Operational Systems track (real-world deployment focus)

Deliverables per Paper

1. One-page summary (markdown) covering:

   • Problem statement: what gap does this paper fill?
   • Key insight/technique: the one idea that makes it work
   • System design: architecture diagram or data flow
   • Evaluation highlights: workload, scale, key numbers
   • Relevance to InferMatrix: which part of our stack could benefit?

2. 5-minute lightning talk (slides) for team sync

3. Action items (optional): concrete experiments or integrations worth trying

Process

• Claim a paper by commenting on this issue (first-come, check the checkboxes)
• Target: 2 papers per intern per week
• Post deep dives as comments on this issue, then we will promote the best ones to docs/deep-dives/
• First sync: EOW Friday — share top 3 findings across all papers

Priority Order (suggested)

1. Prism — multi-model serving with GPU memory ballooning; most directly relevant to our inference routing/placement work
2. Simple is Better — LLM request scheduling with multiplication; scheduling is our core
3. Strata — hierarchical KV cache; our long-context story
4. OpenTela — heterogeneous serving across decentralized resources; matches our multi-backend architecture
5. Kairox — GPU-CPU hybrid inference; our heterogeneous compute story
6. Everything else: pick what interests you

References

• Full proceedings: https://www.usenix.org/system/files/osdi26-full-proceedings.pdf
• Program committee included: Amar Phanishayee (NVIDIA), Deepak Narayanan (NVIDIA), Kan Wu (xAI), Nathan Bronson (OpenAI), Tim Harris (OpenAI)

🤖 Generated with Claude Code

[wtz2333]

Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling

[ShwStone]

Revisiting Pipeline Parallelism for LLM Serving

[zwhzzz0821]

Strata: Hierarchical Context Caching for Long Context Language Model Serving — Stanford, NVIDIA

[s13fmvp]

Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU–GPU Hybrid Design

[mglyn]

No Buffer, No Bottleneck: Efficient Zero-Copy KV Cache Offloading for Long-Context LLMs — UVA

[LJX-xixi]

Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

StriaTrace: Efficient Tracing and Diagnosis for Online LLM Inference

[potatoZhx]

Efficient LLM Serving on Commodity GPU Clusters with Data-Reduced Cross-Instance Orchestration