GPU Usage question and performance data

[FearL0rd]

Thanks this is an interesting project. I'm able to make it RUN but it is not spinning my GPU's cores at 100% and I think it can squeeze more performance there using more RAM and SPIN both cores at 100%

My Specs:

Rysen 36XX 12 Core
Samsung M.2 SSD
128GB RAM
Nvidia V100 32 GB
Nvidia 3090

[FearL0rd]

UPDATE: some tweaks in the parameters, I got 64 tokens in 21.98s (2.91 tok/s)

pulsar: loading /home/cesar/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 20.9s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 857 triples (8.3GB) resident in 11.3s
pulsar: auto budget: 28.1GB VRAM free -> expert cache 24.8GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 38535 slabs in 103.3s
pulsar: prefill 5 tokens in 2.02s
 Paris. The population of France is 65,447,374. The currency used in France is the Euro. The people of France are called French. The national language of France is French. The neighboring countries of France are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, and Switzerland
pulsar: profile: gpu-wait 8.10s, resolve 14.48s (h2d 11.00s, disk/host 3.48s), logits-tail 0.21s over 4875 layer steps
pulsar: 64 tokens in 21.98s (2.91 tok/s), vram cache 55% hits, host cache 97% of remainder, tier 17669 resident slots
pulsar: ids [12089, 13, 576, 7042, 315, 9621, 374, 220, 101411, 11, 19, 101655, 11, 101140, 19, 13, 576, 11408, 1483, 304, 9621, 374, 279, 19424, 13, 576, 1251, 315, 9621, 525, 2598, 8584, 13, 576, 5313, 4128, 315, 9621, 374, 8584, 13, 576, 41282, 5837, 315, 9621, 525, 1597, 82349, 11, 32804, 11, 9851, 11, 15332, 11, 64397, 11, 61119, 11, 17664, 11, 323, 29014]

[FearL0rd]

run after the PR provided + your last update

pulsar: loading /home/cesar/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 20.6s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 857 triples (8.3GB) resident in 12.1s
pulsar: auto budget: 28.1GB VRAM free -> expert cache 24.8GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 39413 slabs in 106.5s
pulsar: prefill 5 tokens in 6.76s
 Paris. The population of France is 65,312,856. The currency used in France is the Euro. The people of France are called French. The national language of France is French. The neighboring countries of France are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, and Switzerland
pulsar: profile: gpu-wait 10.47s, resolve 7.15s (h2d 1.63s, disk/host 5.52s), cpu-lane 0.00s, logits-tail 0.21s over 4875 layer steps
pulsar: 64 tokens in 21.49s (2.98 tok/s), vram cache 72% hits, host cache 92% of remainder, tier 17618 resident slots
pulsar: ids [12089, 13, 576, 7042, 315, 9621, 374, 220, 101411, 11, 18, 98886, 11, 23, 101917, 13, 576, 11408, 1483, 304, 9621, 374, 279, 19424, 13, 576, 1251, 315, 9621, 525, 2598, 8584, 13, 576, 5313, 4128, 315, 9621, 374, 8584, 13, 576, 41282, 5837, 315, 9621, 525, 1597, 82349, 11, 32804, 11, 9851, 11, 15332, 11, 64397, 11, 61119, 11, 17664, 11, 323, 29014]

[giannisanni]

Thanks for the detailed data, this is exactly the kind of box report that helps.

On the GPU utilization: low GPU busy is expected on the streaming giants. Decode is bound by expert bytes, not compute: every token has to pull its routed experts across disk, PCIe, or RAM, and the GPUs spend most of the wall clock waiting on those bytes. Spinning the cores to 100 percent is not the win condition here, feeding them cheaper bytes is.

Which brings me to the thing I would try on your box first: PULSAR_CPU=1. Your log shows host cache at 92 percent of vram-cache misses, which means almost all your expert traffic is RAM-to-GPU uploads over PCIe 3.0 (the V100 tops out around 12 GB/s H2D). The CPU expert lane computes those host-cached experts on the CPU instead, straight out of RAM, and skips the bus entirely. On my box (PCIe 4/5, much smaller RAM) it gained 21 to 41 percent depending on the model; with 128GB RAM and PCIe 3.0 your profile is much more favorable than mine. Your Ryzen 3600 has AVX2, which is all it needs. It is opt-in and numerically equivalent (float-order drift only, same class as multi-GPU tier summing).

It stacks with the improvements in your PR since the lane removes the host-hit uploads and your async H2D accelerates whatever is left. Would be very interested in your before/after numbers with the flag on.

[FearL0rd]

I have tested with the with and without suggested flag and the performance dropped after the latest merge I did from the lasted updates. Investigating now.

WITH PULSAR_CPU=1

pulsar: loading /home/cesar/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 26.7s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 858 triples (8.4GB) resident in 8.2s
pulsar: auto budget: 28.1GB VRAM free -> expert cache 24.8GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 39416 slabs in 126.9s
pulsar: prefill 5 tokens in 2.38s
 Paris. Paris is located on the Seine in the center of the Île-de France region, which has also been the capital region. Its is situated in 48 degrees 35 minutes and 2 seconds north latitude and 0 degrees 35 minutes and 2 seconds west longitude. In 2 degrees 35 minutes
pulsar: profile: gpu-wait 3.27s, resolve 9.17s (h2d 0.18s, disk/host 8.99s), cpu-lane 7.48s, logits-tail 0.14s over 4875 layer steps
pulsar: 64 tokens in 33.14s (1.93 tok/s), vram cache 91% hits, host cache 96% of remainder, tier 13189 resident slots, cpu lane 16743 experts
pulsar: ids [12089, 13, 12089, 374, 7407, 389, 279, 146737, 304, 279, 4126, 315, 279, 58699, 273, 6810, 9621, 5537, 11, 892, 702, 1083, 1012, 279, 6722, 5537, 13, 11440, 374, 29966, 304, 220, 100933, 12342, 220, 100235, 4420, 323, 220, 17, 6486, 10195, 20803, 323, 220, 15, 12342, 220, 100235, 4420, 323, 220, 17, 6486, 9706, 20472, 13, 758, 220, 17, 12342, 220, 100235, 4420]

WITHOUT PULSAR_CPU=1

pulsar: loading /home/cesar/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 26.6s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 858 triples (8.4GB) resident in 15.9s
pulsar: auto budget: 28.1GB VRAM free -> expert cache 24.8GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 39416 slabs in 126.9s
pulsar: prefill 5 tokens in 2.20s
 Paris. The population of France is 65,318,000. The currency used in France is the Euro. The people of France are called French. The neighboring countries of France are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, and Switzerland. France is located in Western Europe. The
pulsar: profile: gpu-wait 11.45s, resolve 10.38s (h2d 1.89s, disk/host 8.49s), cpu-lane 0.00s, logits-tail 0.22s over 4875 layer steps
pulsar: 64 tokens in 25.74s (2.49 tok/s), vram cache 71% hits, host cache 89% of remainder, tier 14825 resident slots
pulsar: ids [12089, 13, 576, 7042, 315, 9621, 374, 220, 101411, 11, 18, 99243, 11, 99752, 13, 576, 11408, 1483, 304, 9621, 374, 279, 19424, 13, 576, 1251, 315, 9621, 525, 2598, 8584, 13, 576, 41282, 5837, 315, 9621, 525, 1597, 82349, 11, 32804, 11, 9851, 11, 15332, 11, 64397, 11, 61119, 11, 17664, 11, 323, 29014, 13, 9621, 374, 7407, 304, 10862, 4505, 13, 576]

[FearL0rd]

I have made some changes and now I'm hitting 64 tokens in 20.94s (3.06 tok/s

pulsar: loading /home/cesar/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 26.6s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 858 triples (8.4GB) resident in 12.0s
pulsar: auto budget: 28.1GB VRAM free -> expert cache 24.8GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 39416 slabs in 128.5s
pulsar: prefill 5 tokens in 1.89s
 Paris. The population of France is 65,633,174. The currency used in France is the Euro. The people of France are called French. The national language of France is French. The neighboring countries of France are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, and Switzerland.
pulsar: profile: gpu-wait 8.53s, resolve 13.09s (h2d 0.51s, disk/host 12.58s), cpu-lane 0.00s, logits-tail 0.21s over 4875 layer steps
pulsar: 64 tokens in 20.94s (3.06 tok/s), vram cache 52% hits, host cache 97% of remainder, tier 14760 resident slots
pulsar: ids [12089, 13, 576, 7042, 315, 9621, 374, 220, 101411, 11, 21, 100702, 11, 127031, 13, 576, 11408, 1483, 304, 9621, 374, 279, 19424, 13, 576, 1251, 315, 9621, 525, 2598, 8584, 13, 576, 5313, 4128, 315, 9621, 374, 8584, 13, 576, 41282, 5837, 315, 9621, 525, 1597, 82349, 11, 32804, 11, 9851, 11, 15332, 11, 64397, 11, 61119, 11, 17664, 11, 323, 29014, 13]

WITH PULSAR_CPU=1 Lose performance

pulsar: loading /home/cesar/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 26.9s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 858 triples (8.4GB) resident in 11.2s
pulsar: auto budget: 28.1GB VRAM free -> expert cache 24.8GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 39416 slabs in 132.3s
pulsar: prefill 5 tokens in 2.43s
 Paris. Paris is located on the Seine River and is divided into two corners, separated by the water of the River Seine, that the capital of the two sides the River Seine, the capital of the two sides the River Seine, the capital of the two sides the River Seine, the capital of the two sides the River
pulsar: profile: gpu-wait 2.92s, resolve 9.54s (h2d 0.08s, disk/host 9.46s), cpu-lane 8.56s, logits-tail 0.14s over 4875 layer steps
pulsar: 64 tokens in 33.28s (1.92 tok/s), vram cache 90% hits, host cache 96% of remainder, tier 13018 resident slots, cpu lane 18280 experts
pulsar: ids [12089, 13, 12089, 374, 7407, 389, 279, 146737, 10943, 323, 374, 17753, 1119, 1378, 23397, 11, 18630, 553, 279, 3015, 315, 279, 10943, 146737, 11, 429, 279, 6722, 315, 279, 1378, 11062, 279, 10943, 146737, 11, 279, 6722, 315, 279, 1378, 11062, 279, 10943, 146737, 11, 279, 6722, 315, 279, 1378, 11062, 279, 10943, 146737, 11, 279, 6722, 315, 279, 1378, 11062, 279, 10943]

I have commit the changes as "perf: protect warm triples and cheapen host LFU eviction" Please review

[giannisanni]

Great data, and two separate things are going on in it. Let me split them.

1. The lane-on output quality is wrong on your build, and that is the important one. Look at your PULSAR_CPU=1 outputs: both degrade into repetition loops ("the capital of the two sides the River Seine, the capital of the two sides..."). The lane cannot do that by design: it computes the same math in a different float order, which can flip a greedy near-tie once, but it cannot produce loops. On my box GLM with the lane on stays fully coherent. Since you are running your PR branch hand-merged with latest main, my prime suspect is the merge: your cross-layer H2D prefetch and my CPU lane both claim experts, and if a prefetched pointer reaches the GPU expert table while the lane also computes that expert, its output gets added twice. That produces exactly this kind of degradation. The ownership rule has to be: lane-claimed experts are excluded from the H2D prefetch, NULL in every GPU pointer table, and their offsets pinned in the host store. This integration is the fiddly part I flagged in the PR review, and I would much rather do it once, properly, in the rebase. The offer stands: say the word and I will push a rebased version of your branch with the interaction wired correctly.

2. Even with a correct merge, your box legitimately wants a different lane policy, so I shipped you a knob. Your triple warm load already gets 91 percent VRAM hits, and the lane's default rule (CPU owns every host-cached expert, even VRAM-resident ones) steals experts your GPU had for free, then chews them on a Ryzen 3600 at about half my per-thread throughput: your cpu-lane wall time was 7.48s. That rule exists because it stabilizes the cache ecology on my 30GB-RAM box where VRAM covers far less. As of the latest main, PULSAR_CPU_STEAL=0 makes the lane leave dev-cache-resident experts to the GPU and only take true upload-bound ones. On your topology I would expect PULSAR_CPU=1 PULSAR_CPU_STEAL=0 to be the right combination, but only after the merge issue from point 1 is fixed, otherwise the quality bug masks everything.

Meanwhile your 3.06 tok/s with your own changes is real progress over the 2.94 in the PR. Once the rebase lands we can measure lane + triples + async H2D together cleanly.

[FearL0rd]

@giannisanni
I switch the cards PCIe now 3090 x16 and V100 x4. I got my best run

scanning GPUs (min 8192 MiB total VRAM)...
hide GPU 0 NVIDIA GeForce GTX 1060 3GB (3072 MiB) — denylist
cand GPU 1 Tesla V100-PCIE-32GB free=32486 MiB PCIe gen2 x16 (score=32)
cand GPU 2 NVIDIA GeForce RTX 3090 free=24107 MiB PCIe gen3 x16 (score=48)

selected topology:
STREAM primary physical GPU 2 NVIDIA GeForce RTX 3090 (free 24107 MiB, PCIe score 48)
ATTN secondary physical GPU 1 Tesla V100-PCIE-32GB (free 32486 MiB)

CUDA_VISIBLE_DEVICES=2,1
PULSAR_GPU=0
PULSAR_ATTN_GPU=1
PULSAR_CACHE_GB=106 (auto: 122G avail - 16G headroom)
PULSAR_ATTN_VRAM_GB=16 (auto: ~32G free on attn → budget 16G stack, leave rest for expert tier)
PULSAR_CPU=1
PULSAR_CPU_STEAL=0
model: /home/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf

physical cards in use:
index, name, memory.total [MiB], memory.free [MiB], pcie.link.gen.current, pcie.link.width.current
1, Tesla V100-PCIE-32GB, 32768 MiB, 32486 MiB, 2, 4
2, NVIDIA GeForce RTX 3090, 24576 MiB, 24107 MiB, 3, 16

pulsar: loading /home/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 25.1s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 1756 triples (17.1GB) resident in 27.0s
pulsar: auto budget: 19.4GB VRAM free -> expert cache 16.1GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 40031 slabs in 124.4s
pulsar: prefill chunk @0 len 5 in 1.73s
pulsar: prefill 5 tokens in 1.73s
Paris. The population of France is 65,821,889. The currency used in France is the Euro. The people of France are called French. The national language of France is French. The neighboring countries of France are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, and Switzerland
pulsar: profile: gpu-wait 4.73s, resolve 3.39s (d2h 0.18s, lists 0.30s, host 2.50s, h2d 0.02s, other 0.39s), cpu-lane 4.25s, logits-tail 0.09s over 4875 layer steps
pulsar: 64 tokens in 19.16s (3.34 tok/s), vram cache 94% hits, host cache 97% of remainder, tier 23508 resident slots, cpu lane 10292 experts
pulsar: ids [12089, 13, 576, 7042, 315, 9621, 374, 220, 101411, 11, 23, 99146, 11, 101252, 24, 13, 576, 11408, 1483, 304, 9621, 374, 279, 19424, 13, 576, 1251, 315, 9621, 525, 2598, 8584, 13, 576, 5313, 4128, 315, 9621, 374, 8584, 13, 576, 41282, 5837, 315, 9621, 525, 1597, 82349, 11, 32804, 11, 9851, 11, 15332, 11, 64397, 11, 61119, 11, 17664, 11, 323, 29014]

[FearL0rd]

@giannisanni setting PULSAR_CPU=12 I crossed (3.69 tok/s)

CUDA_VISIBLE_DEVICES=2,1
PULSAR_GPU=0
PULSAR_ATTN_GPU=1
PULSAR_CACHE_GB=106 (auto: 122G avail - 16G headroom)
PULSAR_ATTN_VRAM_GB=16 (auto: ~32G free on attn → budget 16G stack, leave rest for expert tier)
PULSAR_CPU=12
PULSAR_CPU_STEAL=0
model: /home/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf

physical cards in use:
index, name, memory.total [MiB], memory.free [MiB], pcie.link.gen.current, pcie.link.width.current
1, Tesla V100-PCIE-32GB, 32768 MiB, 32486 MiB, 2, 4
2, NVIDIA GeForce RTX 3090, 24576 MiB, 24107 MiB, 3, 16

pulsar: loading /home/models/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf
pulsar: using CUDA device 0
pulsar: attn weights + KV resident on CUDA device 1
pulsar: loaded in 26.5s (78 layers, 256 experts x top-8)
pulsar: prompt ids [785, 6722, 315, 9621, 374]
pulsar: expert tier on CUDA device 1: 1756 triples (17.1GB) resident in 27.7s
pulsar: auto budget: 19.4GB VRAM free -> expert cache 16.1GB, staging 2.5GB, prefill chunk 256
pulsar: warm start: 40031 slabs in 124.4s
pulsar: prefill chunk @0 len 5 in 1.38s
pulsar: prefill 5 tokens in 1.38s
 Paris. The population of France is 65,447,374. The currency used in France is the Euro. The people of France are called French. The national language of France is French. The neighboring countries of France are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, and Switzerland
pulsar: profile: gpu-wait 4.69s, resolve 3.93s (d2h 0.18s, lists 0.31s, host 3.01s, h2d 0.03s, other 0.40s), cpu-lane 3.34s, logits-tail 0.09s over 4875 layer steps
pulsar: 64 tokens in 17.32s (3.69 tok/s), vram cache 93% hits, host cache 97% of remainder, tier 23093 resident slots, cpu lane 10688 experts
pulsar: ids [12089, 13, 576, 7042, 315, 9621, 374, 220, 101411, 11, 19, 101655, 11, 101140, 19, 13, 576, 11408, 1483, 304, 9621, 374, 279, 19424, 13, 576, 1251, 315, 9621, 525, 2598, 8584, 13, 576, 5313, 4128, 315, 9621, 374, 8584, 13, 576, 41282, 5837, 315, 9621, 525, 1597, 82349, 11, 32804, 11, 9851, 11, 15332, 11, 64397, 11, 61119, 11, 17664, 11, 323, 29014]