docs: add Jetson AGX Xavier MTP speculative decoding benchmarks for Qwen3.6-35B-A3B

benchmark_results/jetson_agx_xavier/FINDINGS.md

@@ -0,0 +1,82 @@
+# Jetson AGX Xavier MTP Benchmark: Q3_K_M vs APEX I-Compact
+
+## Device
+
+| Spec | Value |
+|---|---|
+| Device | NVIDIA Jetson AGX Xavier (MiiVii, Tegra194) |
+| GPU | Volta SM72, 512 CUDA cores, 7.6 TFLOPS FP16 |
+| Memory | 32GB unified (CPU+GPU), ~135 GB/s bandwidth |
+| Power | 30W |
+| OS | Ubuntu 20.04 aarch64 |
+| CUDA | 11.4 |
+| Software | llama.cpp master (SHA 571d0d54), built with CUDA |
+
+## Models Tested
+
+Both quantizations of `Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved` (the uncensored fine-tune that preserves all 20 MTP layers):
+
+- **Q3_K_M (K-quant, bartowski)**: 17.3 GB — standard K-quant baseline
+- **APEX I-Compact (SC117/cvgro)**: 17.0 GB — APEX layer-wise precision gradient (5 + 5 edge layers at higher precision, middle 30 layers compressed)
+- ~~APEX I-Mini~~: Not available at time of writing (404 on both SC117 and cvgro repos). Original PR data retained for reference only.
+
+## Methodology
+
+Each configuration tested with **5 runs** (128 tokens each, fixed prompt, seed=42). Results reported as **mean ± std dev** over 5 runs. Sequential test order: Q3_K_M baseline → Q3_K_M MTP → APEX baseline → APEX MTP (same session, no reboot between config changes).
+
+> **Note on baseline drift**: The original PR single-run data recorded Q3_K_M baseline at 15.96 t/s. Our 5-run mean is 19.32 t/s — a ~21% shift attributed to environmental differences (thermal state, system load). This discrepancy does not affect within-session comparisons (baseline vs MTP for the same model), but cross-session speedup ratios are unreliable.
+
+## Baseline Results (no speculative decoding)
+
+| Quant | Size | Mean ± σ | CV | vs Q3_K_M |
+|---|---|---|---|---|
+| Q3_K_M | 17.3 GB | **19.32 ± 0.02 t/s** | 0.1% | baseline |
+| **APEX I-Compact** | 17.0 GB | **19.97 ± 0.05 t/s** | 0.2% | **+3.4%** |
+
+Both quantizations show extremely low variance (≤0.2% CV), confirming test reproducibility. APEX I-Compact's **+3.4% baseline advantage** over Q3_K_M is small but consistent (all 5 runs above Q3_K_M's max). This may reflect APEX's selective precision allocation improving compute efficiency on memory-bandwidth-bound AGX Xavier.
+
+> Original PR reported Q3_K_M at 15.96 t/s and APEX I-Compact at 15.97 t/s (essentially tied). Our multi-run data shows a small but reproducible APEX advantage.
+
+> APEX I-Mini original PR baseline: 14.67 t/s (single run, not verified).
+
+## MTP Speculative Decoding Results (`--spec-type draft-mtp --spec-draft-n-max 2`)
+
+| Quant | Mean ± σ | CV | Speedup vs baseline | MTP Accept Rate* |
+|---|---|---|---|---|
+| Q3_K_M | **22.47 ± 0.40 t/s** | 1.8% | **+16.3%** | 97.7% |
+| **APEX I-Compact** | **24.85 ± 0.07 t/s** | 0.3% | **+24.4%** | 69.8% |
+
+> *Accept rates from original PR (single run). Q3_K_M speedup of 16.3% vs our baseline (original PR: +45% vs 15.96 t/s). APEX speedup of 24.4% (original PR: +19.0% vs 15.97 t/s).
+
+**Key observation**: APEX I-Compact achieves **higher absolute MTP throughput (24.85 t/s) and higher speedup (+24.4%)** than Q3_K_M MTP (22.47 t/s, +16.3%), despite a significantly lower acceptance rate (69.8% vs 97.7%). The directly observed result stands on its own — the acceptance rates are carried over from the original PR (different session, ~21% Q3 baseline drift), so no causal formula can be reliably derived from cross-session measurements.
+
+## Key Finding
+
+On this Jetson AGX Xavier test system, **APEX I-Compact outperforms Q3_K_M in both baseline (+3.4%) and MTP (+10.6% absolute)** throughput (directly observed, same test session).
+
+### Practical implications
+
+| Metric | Q3_K_M | APEX I-Compact | Winner |
+|---|---|---|---|
+| Baseline t/s | 19.32 | 19.97 | **APEX** (+3.4%) |
+| MTP t/s | 22.47 | 24.85 | **APEX** (+10.6%) |
+| Speedup ratio | +16.3% | +24.4% | **APEX** |
+| Model size | 17.3 GB | 17.0 GB | **APEX** (−0.3 GB) |
+| Accept rate | 97.7% | 69.8% | Q3_K_M |
+| TPS stability (CV) | 1.8% | 0.3% | **APEX** |
+
+## Caveats
+
+1. **Single device (N=1)**: Results are device-specific and may not generalize.
+2. **Single prompt, fixed length**: Only tested with one prompt (180 chars) and 128 output tokens.
+3. **Thermal ordering**: APEX was tested after Q3_K_M — the device may have been in a different thermal state.
+4. **Accept rate not independently measured**: Values from original PR (different session, ~21% Q3 baseline drift). Accept rates on our hardware may differ significantly — direct measurement would be needed for a proper causal model.
+5. **No APEX I-Mini**: The 13.3 GB variant is no longer available, limiting the comparison to similar-size models only.
+6. **Causal mechanism speculative**: The link between APEX's layer-wise precision gradient and accept rate reduction is inferred, not experimentally validated.
+
+## Suggested Next Steps
+
+1. **Direct acceptance rate measurement** via llama.cpp's `--print-acceptance` or per-token logprobs comparison
+2. **Test APEX I-Balanced/I-Quality** on devices with ≥48 GB unified memory to see if gentler compression preserves accept rates
+3. **Cross-validate with different prompts and sequence lengths**
+4. **Run reverse order** (APEX first, Q3_K_M second) to isolate thermal effects

benchmark_results/jetson_agx_xavier/mtp_results.txt

@@ -0,0 +1,28 @@
+# MTP Speculative Decoding Benchmarks on NVIDIA Jetson AGX Xavier
+# Device: Jetson AGX Xavier (Volta, 512-core, 32GB unified memory, ~135 GB/s, 30W)
+# Software: llama.cpp master (SHA 571d0d54), CUDA 11.4, Ubuntu 20.04 aarch64
+# MTP config: --spec-type draft-mtp --spec-draft-n-max 2
+# Benchmark: 128 tokens generated after fixed prompt (180 chars), 5 runs each, seed=42
+# Base model: Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved
+#
+# Format:
+# model  size_gb  mean_tps  std_tps  min_tps  max_tps  n_runs
+# For single-run values (original PR): mean_tps is the single value, n_runs=1
+# tps = tokens per second (timings.predicted_per_second from API)
+#
+# --- All 5-run data collected 2026-07-24, same prompt+seed, sequential runs ---
+#
+# Q3_K_M (bartowski), baseline (no speculative decoding)
+Q3_K_M_baseline  17.3  19.32  0.02  19.29  19.34  5
+# Q3_K_M (bartowski), MTP (--spec-type draft-mtp --spec-draft-n-max 2)
+Q3_K_M_mtp_n2    17.3  22.47  0.40  21.76  22.73  5
+#
+# APEX I-Compact (SC117/cvgro), baseline (no speculative decoding)
+APEX_I-Compact_baseline  17.0  19.97  0.05  19.92  20.02  5
+# APEX I-Compact (SC117/cvgro), MTP (--spec-type draft-mtp --spec-draft-n-max 2)
+APEX_I-Compact_mtp_n2    17.0  24.85  0.07  24.76  24.91  5
+#
+# --- Original PR single-run data (kept for reference) ---
+# Note: Q3_K_M baseline 15.96 t/s (single run) vs our 19.32 t/s (5-run mean)
+# Baseline drift ~21% attributed to thermal/system load differences
+# MTP accept rates from original PR (different session): Q3_K_M: 97.7%, APEX I-Compact: 69.8%, APEX I-Mini: 70.5%