Add SGLang Qwen3-8B torch.profiler harness and per-component latency classifier

.github/workflows/ut.yml

@@ -51,7 +51,7 @@ jobs:
 
       - name: Build
         run: |
-          apt-get update && apt-get install -y lcov
+          apt-get update && apt-get install -y --no-install-recommends lcov
           mkdir build && cd build
           CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Debug"
           if [ "${{ matrix.platform }}" = "rocm" ]; then

examples/qwen3/PROFILE.md

@@ -0,0 +1,60 @@
+# Qwen3-8B Per-Component Latency Profile
+
+Per-component latency breakdown of SGLang Qwen3-8B (TP=8, batch=1) on
+8×A100-80GB.  Profiled with `torch.profiler` via `profile_sglang.py`.
+
+## Configuration
+
+| Parameter       | Value                                   |
+|-----------------|-----------------------------------------|
+| Model           | Qwen/Qwen3-8B                           |
+| SGLang image    | lmsysorg/sglang:v0.4.6.post1-cu124      |
+| Hardware        | 8×A100-80GB (SXM4)                      |
+| TP              | 8                                        |
+| Batch           | 1                                        |
+| Prompt length   | 2048 tokens                              |
+| Generate length | 128 tokens                               |
+| Profiler        | torch.profiler (wait=2, warmup=1, active=3) |
+
+## Prefill Phase (prompt=2048, max_new_tokens=1)
+
+| Component      | Kernel time (ms) | % of total | ARK target | Q-item |
+|----------------|-----------------|------------|------------|--------|
+| gemm_mlp       | TBD             | TBD        | TBD        | Q5     |
+| gemm_attention | TBD             | TBD        | TBD        | Q4     |
+| attention      | TBD             | TBD        | TBD        | Q6     |
+| norms_rope     | TBD             | TBD        | TBD        | Q7     |
+| nccl           | TBD             | TBD        | TBD        | Q8     |
+| embed_lm_head  | TBD             | TBD        | TBD        | —      |
+| other          | TBD             | TBD        | TBD        | —      |
+| **Total**      | **TBD**         | **100%**   |            |        |
+
+## Decode Phase (prompt=2048, max_new_tokens=128)
+
+| Component      | Kernel time (ms) | % of total | ARK target | Q-item |
+|----------------|-----------------|------------|------------|--------|
+| gemm_mlp       | TBD             | TBD        | TBD        | Q5     |
+| gemm_attention | TBD             | TBD        | TBD        | Q4     |
+| attention      | TBD             | TBD        | TBD        | Q6     |
+| norms_rope     | TBD             | TBD        | TBD        | Q7     |
+| nccl           | TBD             | TBD        | TBD        | Q8     |
+| embed_lm_head  | TBD             | TBD        | TBD        | —      |
+| other          | TBD             | TBD        | TBD        | —      |
+| **Total**      | **TBD**         | **100%**   |            |        |
+
+## Q4–Q8 Re-ranking
+
+Re-rank Q4–Q8 by descending kernel time once numbers land.  The component
+that consumes the most GPU time gets highest optimization priority.
+
+| Priority | Q-item | Component | Prefill % | Decode % |
+|----------|--------|-----------|-----------|----------|
+| 1        | TBD    | TBD       | TBD       | TBD      |
+| 2        | TBD    | TBD       | TBD       | TBD      |
+| 3        | TBD    | TBD       | TBD       | TBD      |
+| 4        | TBD    | TBD       | TBD       | TBD      |
+| 5        | TBD    | TBD       | TBD       | TBD      |
+
+Numbers and final ordering filled out-of-band after profiling on
+`mscclpp-a100-dev`.  See [reproduce_profile.md](bench/reproduce_profile.md)
+for repro steps.

examples/qwen3/bench/analyze_profile.sh

@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+# Analyze a torch.profiler Chrome-trace JSON for Qwen3-8B per-component budget.
+#
+# Usage:
+#   bash analyze_profile.sh <trace.json> [--tp 8]
+#
+# Steps:
+#   1. Run trace_analyzer.py (torch-profiler skill) for kernel/comm/gap sections.
+#   2. Run classify_kernels.py to produce the per-component latency budget.
+#
+# Prerequisites:
+#   - Python 3.10+
+#   - trace_analyzer.py path set via TRACE_ANALYZER env var, or auto-detected
+#     from the torch-profiler skill directory.
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+# --- Arguments ----------------------------------------------------------------
+
+if [ $# -lt 1 ]; then
+    echo "Usage: $0 <trace.json> [--tp 8]" >&2
+    exit 1
+fi
+
+TRACE_FILE="$1"
+shift
+
+TP=8
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --tp) if [ $# -lt 2 ]; then echo "Error: --tp requires a value" >&2; exit 1; fi; TP="$2"; shift 2 ;;
+        *) echo "Unknown argument: $1" >&2; exit 1 ;;
+    esac
+done
+
+if [ ! -f "${TRACE_FILE}" ]; then
+    echo "Error: trace file not found: ${TRACE_FILE}" >&2
+    exit 1
+fi
+
+# --- Locate trace_analyzer.py -------------------------------------------------
+
+TRACE_ANALYZER="${TRACE_ANALYZER:-}"
+if [ -z "${TRACE_ANALYZER}" ]; then
+    # Try common locations
+    for candidate in \
+        "${SCRIPT_DIR}/../../../.pi/skills/torch-profiler/scripts/trace_analyzer.py" \
+        "${HOME}/.pi/skills/torch-profiler/scripts/trace_analyzer.py"; do
+        if [ -f "${candidate}" ]; then
+            TRACE_ANALYZER="${candidate}"
+            break
+        fi
+    done
+fi
+
+# --- Step 1: trace_analyzer.py (if available) ---------------------------------
+
+OUTPUT_DIR="$(dirname "${TRACE_FILE}")/analysis"
+mkdir -p "${OUTPUT_DIR}"
+
+if [ -n "${TRACE_ANALYZER}" ] && [ -f "${TRACE_ANALYZER}" ]; then
+    echo "=== Trace Analyzer: kernel breakdown ==="
+    python3 "${TRACE_ANALYZER}" "${TRACE_FILE}" --section kernels --top-n 30 \
+        | tee "${OUTPUT_DIR}/kernels.txt"
+    echo ""
+
+    echo "=== Trace Analyzer: communication ==="
+    python3 "${TRACE_ANALYZER}" "${TRACE_FILE}" --section comm \
+        | tee "${OUTPUT_DIR}/comm.txt"
+    echo ""
+
+    echo "=== Trace Analyzer: GPU gaps ==="
+    python3 "${TRACE_ANALYZER}" "${TRACE_FILE}" --section gaps \
+        | tee "${OUTPUT_DIR}/gaps.txt"
+    echo ""
+
+    echo "=== Trace Analyzer: full summary ==="
+    python3 "${TRACE_ANALYZER}" "${TRACE_FILE}" --full \
+        -o "${OUTPUT_DIR}/full_analysis.md"
+    echo "Full analysis saved to ${OUTPUT_DIR}/full_analysis.md"
+    echo ""
+else
+    echo "Warning: trace_analyzer.py not found. Set TRACE_ANALYZER env var." >&2
+    echo "Skipping trace_analyzer.py step." >&2
+    echo ""
+fi
+
+# --- Step 2: per-component classification -------------------------------------
+
+echo "=== Per-component classification (TP=${TP}) ==="
+python3 "${SCRIPT_DIR}/classify_kernels.py" "${TRACE_FILE}" --tp "${TP}" \
+    | tee "${OUTPUT_DIR}/component_budget.md"
+
+echo ""
+echo "Analysis complete. Results in: ${OUTPUT_DIR}/"