Skip to content

Add SGLang Qwen3-8B torch.profiler harness and per-component latency classifier#264

Closed
chhwang wants to merge 4 commits into
mainfrom
qwen3-q2-profile
Closed

Add SGLang Qwen3-8B torch.profiler harness and per-component latency classifier#264
chhwang wants to merge 4 commits into
mainfrom
qwen3-q2-profile

Conversation

@chhwang

@chhwang chhwang commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Add SGLang Qwen3-8B torch.profiler harness and per-component latency classifier

Q2 of the Qwen3-on-ARK initiative: instrument SGLang's Qwen3-8B forward
with torch.profiler to produce a ranked per-component latency budget for
both prefill and decode phases.

New files under examples/qwen3/:

  • bench/profile_sglang.py — HTTP client that drives SGLang's
    /start_profile//stop_profile endpoints for prefill (prompt=2048,
    max_new_tokens=1) and decode (prompt=2048, max_new_tokens=128) phases.
    Configurable schedule (wait=2, warmup=1, active=3), record_shapes,
    with_stack.
  • bench/classify_kernels.py — classifies Chrome-trace kernel events into
    seven component buckets (attention, gemm_attention, gemm_mlp, nccl,
    norms_rope, embed_lm_head, other) using name-pattern matching and
    shape-based GEMM disambiguation for Qwen3-8B at TP=8.
  • bench/analyze_profile.sh — runs the torch-profiler skill's
    trace_analyzer.py then classify_kernels.py, writing results to an
    analysis/ directory.
  • bench/test_profile_sglang.py — 40+ CPU-only unit tests covering kernel
    classification, GEMM shape disambiguation, profiler config construction,
    argument parsing, and budget formatting.
  • PROFILE.md — template with per-component tables for prefill and decode;
    numbers filled out-of-band after profiling on 8×A100.
  • bench/reproduce_profile.md — step-by-step reproduction instructions.

CI fix: add --no-install-recommends to apt-get install lcov in
.github/workflows/ut.yml to prevent OOM kill (exit 137) from
fontconfig-config dependency tree on the self-hosted CUDA runner.

…filer for both prefill and decode phases; write PROFILE.md with a ranked per-component latency budget that re-ranks Q4–Q8.
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.70%. Comparing base (c257202) to head (427d7d1).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #264   +/-   ##
=======================================
  Coverage   85.70%   85.70%           
=======================================
  Files         129      129           
  Lines        6457     6457           
=======================================
  Hits         5534     5534           
  Misses        923      923           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ark-dev and others added 3 commits June 12, 2026 05:48
…filer for both prefill and decode phases; write PROFILE.md with a ranked per-component latency budget that re-ranks Q4–Q8.
The CUDA UnitTest job OOM-kills during 'apt-get install lcov' because
fontconfig-config and its recommended packages exhaust container memory.
Adding --no-install-recommends avoids pulling those heavy deps.
…onent latency budget for prefill and decode, write PROFILE.md with ranked breakdown and targets
@chhwang chhwang changed the title ark-dev: Implement Q2: profile SGLang Qwen3-8B forward with torch.profiler for both prefill and decode phases; write PROFILE.md with a ranked per-component latency budget that re-ranks Q4–Q8. Add SGLang Qwen3-8B torch.profiler harness and per-component latency classifier Jun 12, 2026
@chhwang

chhwang commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the SGLang baseline/profiling harness is kept local for evaluation only and will not be merged into this repo. Only the ARK implementation lands here.

@chhwang chhwang closed this Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant