Device: NVIDIA Jetson AGX Xavier (Volta 512-core, ~135 GB/s unified memory, 32 GB RAM, 30W TDP)
Context: I'm running the Qwen3.6-35B-A3B-Native-MTP-Preserved model on llama.cpp (master) via the Jetson platform. Currently getting 23.14 t/s with MTP (spec-max=2, 97.7% acceptance) on the original Q3_K_M quant (17.3 GB).
Question: I just downloaded the APEX-MTP-I-Mini (13.3 GB) and APEX-MTP-Compact (16.1 GB) variants. On a Jetson Xavier bandwidth-limited setup (~135 GB/s), the I-Mini should theoretically do ~28 t/s (17.3/13.3 * 23.14). Has anyone tested APEX quantized models on similar embedded/edge hardware? Any known gotchas with llama.cpp + APEX on UMA/integrated GPU systems?
Plan: I'll benchmark both and report back here with numbers. Happy to provide any calibration/speed data that would help the project.
Device: NVIDIA Jetson AGX Xavier (Volta 512-core, ~135 GB/s unified memory, 32 GB RAM, 30W TDP)
Context: I'm running the Qwen3.6-35B-A3B-Native-MTP-Preserved model on llama.cpp (master) via the Jetson platform. Currently getting 23.14 t/s with MTP (spec-max=2, 97.7% acceptance) on the original Q3_K_M quant (17.3 GB).
Question: I just downloaded the APEX-MTP-I-Mini (13.3 GB) and APEX-MTP-Compact (16.1 GB) variants. On a Jetson Xavier bandwidth-limited setup (~135 GB/s), the I-Mini should theoretically do ~28 t/s (17.3/13.3 * 23.14). Has anyone tested APEX quantized models on similar embedded/edge hardware? Any known gotchas with llama.cpp + APEX on UMA/integrated GPU systems?
Plan: I'll benchmark both and report back here with numbers. Happy to provide any calibration/speed data that would help the project.