LLM Inference Papers

Table of Contents

• LLM Serving Systems
• LLM Inference
• Multi-Token Prediction
• Hallucinations
• New Architectures
• MoE
• Hardware-Aware Algorithm Design
• Parameter Efficient Fine Tuning Techniques
• Training Papers
• Quantization and Compression
• Communication Collectives

LLM Serving Systems

• Clipper

  • Conference: NSDI'2017
  • Link: Clipper: A Low-Latency Online Prediction Serving System

• InferLine

  • Conference: ACM'2018
  • Link: InferLine: ML Inference Pipeline Composition Framework

• Orca

  • Conference: OSDI'2022
  • Link: Orca: Differentially Private Data Release for Large-Scale ML Models
  • Orca tackles inefficiencies in traditional LLM inference caused by request-level scheduling, where a batch must wait for all requests to reach the EOS token before returning results, leading to increased latency for shorter requests. Orca introduces iteration-level scheduling, returning tokens at each generation step, thus reducing wait times.Summary

• FlexGen

  • Conference: ICLR'2023
  • Link: FlexGen: High-Throughput Generation with Flexible Large Language Models
  • Code

• vLLM

  • Conference: ACM'2023
  • Link: vLLM: A High-Throughput Transformer-Based Language Model Inference Engine
  • Code
  • The paper introduces a new system for efficient serving of large language models using a novel technique called "paged attention" to improve memory management and throughput during inference.

• Sarathi-serve

  • Conference: OSDI'2024
  • Link: Sarathi-serve: High Performance Serving for Deep Learning Models
  • Code

LLM Inference

• Sarathi

  • Link: SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills

• Flash Decode

  • Link: FlashDecoding++: Faster Large Language Model Inference on GPUs

• Splitwise

  • Conference: IEEE'2023
  • Link: Splitwise: Efficient generative LLM inference using phase splitting

• SGLang

  • Link: Efficiently Programming Large Language Models using SGLang

• DistServe

  • Conference: OSDI'2024
  • Link: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

• Fairness

  • Usenix'2020
  • Link: Fairness in Serving Large Language Models
  • Code

• APIServe

  • Link: APIServe: Efficient API Support for Large-Language Model Inferencing

• LayerSkip

  • Link: LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding

• Parrot

  • Conference: OSDI'2024
  • Link: Parrot: Efficient Serving of LLM-based Applications with Semantic Variable

Multi-Token Prediction

• Speculative Decoding

  • Conference: ICML'2023
  • Link: Fast Inference from Transformers via Speculative Decoding

• Medusa

  • Conference: ICML'2024
  • Link: Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
  • Code

• Lookahead Decoding

  • Conference: ICML'2024
  • Link: Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
  • Code

• Multi-Token Prediction

  • Conference: ICML'2024
  • Link: Better & Faster Large Language Models via Multi-token Prediction

Multiple LLM and multiple LoRa serving

• s-LoRa

  • Conference: MLSys'2024
  • Link: S-LoRA: Serving Thousands of Concurrent LoRA Adapters
  • Code

• MuxServe

  • Conference: ICML'2024
  • Link: MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
  • Code

Hallucinations

• DoLa
  • Conference: ICLR'2024
  • Link: DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
  • Code

New Architectures

• Mamba

  • Link: Mamba: Linear-Time Sequence Modeling with Selective State Spaces

• Jamba

  • Conference: NeurIPS'2023
  • Link: Jamba: A Hybrid Transformer-Mamba Language Model

MoE

• DeepSpeed-MoE

  • Conference: ICML'2022
  • Link: DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

• Tutel

  • Conference: MLSys'2023
  • Link: Tutel: Adaptive Mixture-of-Experts at Scale

• MegaBlocks

  • Conference: MLSys'2023
  • Link: MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

• Gshard

  • Conference: NeurIPS'2022
  • Link: GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

• Towards MoE Deployment

  • Conference: IEEE'2023
  • Link: Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference

• SwapMoE

  • Conference: ACL'2024
  • Link: SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

Hardware-Aware Algorithm Design

• Multi Query Attention

  • Link: Fast Transformer Decoding: One Write-Head is All You Need

• FlashAttention

  • Conference: NeurIPS'2022
  • Link: FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

• FlashAttention 2

  • Link: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

• Grouped Query Attention

  • Conference: EMNLP'2023
  • Link: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

• FlashAttention 3

  • Link: FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
  • Code

Parameter Efficient Fine Tuning Techniques

• LoRa

  • Conference: ICLR'2022
  • Link: https://arxiv.org/abs/2106.09685

• QLoRa

  • Conference: NeurIPS'2023
  • Link: https://arxiv.org/abs/2305.14314

• Soft Prompt Tuning

  • Conference: NeurIPS'2021
  • Link: https://arxiv.org/abs/2104.08691

• Prefix Tuning

  • Conference: ACL'2021
  • Link: https://arxiv.org/abs/2101.00190

• DoRa

  • Conference: ICLR'2024
  • Link: https://arxiv.org/abs/2402.09353

• IA3

  • Conference: NeurIPS'2022
  • Link: https://arxiv.org/pdf/2205.05638

Training Papers

• GPipe

  • Conference - NeurIPS'2019
  • Link: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

• Megatron LM

  • Conference: ICML'2020
  • Link: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

• ZeRO

  • Link: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

• PyTorch Distributed

  • Link: PyTorch Distributed: Experiences on Accelerating Data Parallel Training

• MegatronLM

  • Conference: ACM'2021
  • Link: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
  • Code

• Varuna

  • Link: Varuna: Scalable, Low-cost Training of Massive Deep Learning Models
  • Code

• Alpa

  • Conference: OSDI'2022
  • Link: Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

• Llama 3.1

  • Link: The Llama 3 Herd of Models

• LLM Training Puzzles

Tiny ML

Quantization and Compression

• Deep Compression

  • Conference: ICLR'2015
  • Link: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding
  • Deep Compression aims to reduce the memory and energy requirements of large neural networks by compressing them to fit on-chip SRAM rather than relying on costly DRAM.Summary

• Linear Quantization

  • Conference: IEEE'2018
  • Link: Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

• GPTQ

  • Conference: ICLR'2023
  • Link: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

• SmoothQuant

  • Conference: ICML'2023
  • Link: SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

• DynaQuant

  • Conference: ACM'2023
  • Link: DynaQuant: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

Communication Collectives

• Collective Communication Review

• Link: An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collectives

  • Conference: IEEE'2020

• Link: Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads

  • Conference: ACM'2022

• Synthesizing Optimal Collective Algorithms

  • Conference: ACM'2021
  • Link: Synthesizing Optimal Collective Algorithms

• MSCCLang

  • Conference: ACM'2023
  • Link: MSCCLang: Microsoft Collective Communication Language

TACCL

• NSDI'2023
• Link: TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches

Contribution

Contribute to Our ML Systems Reading List

We’re building a comprehensive and up-to-date GitHub repository of the most impactful papers in Machine Learning Systems (MLSys), and we need your help to keep it growing! Whether you’ve come across a groundbreaking paper, an insightful study, or an innovative idea in the MLSys domain, your contributions can make a difference.

Why contribute?

• Stay Engaged: Sharing papers not only helps others but also keeps you engaged with the latest research trends and developments in MLSys.
• Community Growth: Your contributions foster a collaborative learning environment, helping fellow researchers, engineers, and enthusiasts discover valuable resources.
• Recognition: Each contribution will be attributed to you, allowing you to build a visible presence in the MLSys community.

How to contribute?

1. Find a Paper: Identify a paper that you believe adds value to the repository.
2. Fork the Repository: Create a fork of the repository to make your changes.
3. Add the Paper: Include the paper in the appropriate section of the reading list, following the existing format.
4. Open a Pull Request: Once you’ve added the paper, open a Pull Request (PR) with a brief description of why you think the paper is important.
5. Engage: Engage with any feedback on your PR, making revisions as necessary.

By contributing, you’re not just adding a link—you’re helping to shape the learning resources for the next generation of MLSys practitioners. Let’s work together to make this repository the go-to place for anyone interested in the cutting-edge of ML Systems!

---