Skip to content

Releases: casys-kaist/LLMServingSim

Release list

Release v1.1.0

Release v1.1.0 Latest
Latest

Choose a tag to compare

View all tags
@JaehongCS20 JaehongCS20 released this 30 Apr 17:50
v1.1.0
3012eb1

Highlights

  • vLLM-based layerwise profiler — replaces the old PyTorch Profiler + scikit-learn pipeline with direct layerwise_profile() via a worker extension class, capturing per-layer CUDA kernel timings from real vLLM execution paths
  • End-to-end bench validation suite (python -m bench) — replays workloads through a real vLLM AsyncLLM engine and compares against simulator output with throughput, latency-CDF, and running/waiting plots
  • DP+EP MoE support — Data Parallel + Expert Parallel with ASTRA-Sim ALLTOALL synchronization via involved_dim dimension scoping on a 2D [tp_size, dp_group_size] topology
  • Agentic session support — closed-loop workloads (SWE-bench, tool-calling agents) with dependency chains via sub_requests[] and tool_duration_ns
  • Skew profiling + 5-axis alpha fit — bimodal decode batches measure FlashAttention tile-padding / SM-imbalance costs; the 5-axis weighted-LS fit recovers the penalty the uniform attention grid can't see (test p50/p90 ≈ 2.7% / 14.8%)
  • Unified 4D attention profiling — single attention.csv over prefill_chunk × kv_prefill × n_decode × kv_decode matching vLLM's chunked-prefill scheduler
  • FP8 KV cache support in both profiler and simulator
  • Chunked prefill enabled by default (matching vLLM v1), with prefix caching compatibility
  • Workload generators (python -m workloads.generators) — multi-turn ShareGPT parser with tokenizer-only or vLLM-batched modes
  • Rich-backed logger shared across simulator, profiler, and bench

What's Changed

Added

  • New vLLM-based layerwise profiler (profiler/) replacing the old llm_profile/ module
  • Unified 4D attention profiling (attention.csv)
  • Skew profiling + 5-axis alpha fit for heterogeneous-decode attention
  • Data-derived bucket axes for the skew fit
  • Per-axis skew density knobs: SKEW_N_FACTOR / SKEW_PC_FACTOR / SKEW_KP_FACTOR / SKEW_KVS_FACTOR
  • Per-TP skew_fit.csv file spills the full per-bucket alpha table out of meta.yaml
  • RTXPRO6000 (NVIDIA RTX PRO 6000 Blackwell) hardware support
  • DP+EP support with ASTRA-Sim ALLTOALL synchronization via involved_dim dimension scoping
  • Wave synchronization for DP groups
  • Agentic session support for closed-loop workloads (SWE-bench, tool-calling agents)
  • --num-reqs CLI argument (replaces --num-req), default changed to 0 (load all)
  • Qwen3-32B and Qwen3-30B-A3B-Instruct-2507 model configs with explicit head_dim support
  • FP8 KV cache simulation and profiling support (--kv-cache-dtype fp8)
  • Chunked prefill support (enabled by default, matching vLLM v1)
  • Prefix cache lock tracking for multi-chunk prefill
  • End-to-end vLLM benchmark + simulator validation suite (bench/)
  • Workload generators (workloads/generators/)
  • Rich-backed logger shared between simulator, profiler, and bench
  • Non-Docker vLLM installer (scripts/install-vllm.sh)

Fixed

  • Skew sweep feasibility filter relaxed from >= to > for n = MSQ cases
  • Missing prefix_match call on non-chunked prefill path
  • Prompt throughput now includes prefix cache hit tokens
  • Prefix cache is_init / lock_prefix fixes for full prefix hits
  • MoE expert latency: per-EP-rank instead of aggregated (was 2x overestimate)
  • MoE weight sharding uses ep_size (not tp_size)
  • system.json collective implementations match topology dimensions
  • DP group termination: instances wait for all members
  • argparse allow_abbrev=False
  • Missing return parser.parse_args() in legacy profiler

Changed

  • --fp → --dtype, --gen → --skip-prefill
  • --request-routing-policy default: RR → LOAD
  • --expert-routing-policy FAST → COPY
  • Cluster config: npu_num/npu_group → tp_size/pp_size/ep_size/dp_group
  • MoE modeling: per-EP-rank latency lookup with even expert-to-rank partitioning
  • Trace comm_type supports dimension scoping (ALLREDUCE:1,0, ALLTOALL:0,1)
  • Latency lookup extrapolates beyond profiled range instead of clamping
  • Profiler rewritten to direct vLLM layerwise_profile() approach
  • Layer names unified between profiler and simulator
  • memory_model.py uses explicit head_dim and q_dim/kv_dim
  • trace_generator.py rewritten with composable helpers
  • Directory restructuring: cluster_config/ → configs/cluster/, model_config/ → configs/model/, dataset/ → workloads/, inference_serving/ → serving/, etc.

Removed

  • internal/, evaluation/ directories
  • --enable-attn-prediction flag and scikit-learn predictor
  • --fp, --gen, --num-req flags
  • Analytical ALLTOALL workaround functions

Full Changelog: CHANGELOG.md

Assets 2
Loading