A CUDA fallback is a property of the call, not of the tensor

[ZacharyZcR]

Closes the permanence half of #767. Your framing there, which this takes literally:

A one-off allocation failure taking a device out for good, with no way back short of a restart, is the actual defect. The 2 GB per-device headroom being too small for a realistic prompt is what triggers it; the permanence is what makes it bad.

The mistake in the old code

if (coli_cuda_matmul(...)) return;
w->cuda_failed = 1;        /* one failure, CPU for the rest of the run */

The failure this guards is almost always a scratch cudaMalloc under memory pressure, and that pressure scales with S. A boolean cannot express that. It records "this tensor is broken" when what actually happened is "this width did not fit" — so a prefill chunk at S=512 condemns decode at S=1, on a card that has plenty of room by then.

The fix: record the width

if (coli_cuda_matmul(...)) { w->cuda_fail_s = 0; return; }   /* it fits again */
w->cuda_fail_s = S;

and admit a call when !w->cuda_fail_s || S < w->cuda_fail_s.

That is the whole change. What falls out of it:

prefill OOMs at S=512, decode runs at S=1	decode is admitted, succeeds, and clears the mark — no restart, which is the reported symptom
genuine device fault	fails at S=1, records 1, S < 1 is never true → never retried. The old permanent behaviour, as a special case of the general rule rather than a branch of its own
repeated pressure at shrinking widths	each failure narrows the gate monotonically; it converges instead of oscillating
wider call arrives after a failure	not retried — a larger S can only need more scratch

No retry counter, no threshold, no generation number, no error-code classification, and no change to the DLL ABI — cuda_failed had exactly four references in the tree.

The notices were lying after this change

They said disabled and this run did NOT use the GPU for them. Both become false once a fallback is per-width, so they now describe what actually happens and state that narrower calls still try the GPU — decode can recover even when prefill does not. That is also the "say it out loud" item from the same comment: the previous text named the mechanism, not the consequence.

What this does not do

Your third item — auto reserving a constant 2 GB where the requirement scales with S — is not addressed here. That is the trigger, and it wants a scratch model plus measurement on real hardware. This PR only stops the trigger from being permanent. #687, #766 and #759 are also untouched.

Verification, and its limit

• CPU build clean, -DCOLI_CUDA syntax build clean, zero warnings
• make test-c passes

I have no CUDA device, so the runtime behaviour is unverified. Triggering this path needs a real OOM. The change is a pure branch-condition change on the C side and I have reasoned through the cases above, but reasoning is not a measurement and I would rather say so than imply otherwise.

@ThefloorMiner — you reported #767 and have the 4× A6000. The check is the original repro: CUDA_EXPERT_GB=auto, a ~7,000-token prompt, then confirm decode returns to the GPU instead of staying on CPU until restart. If it does not, this is wrong and I would like to know.

[JustVugg]

The change itself is right, and the framing is the part I would keep even if the code changed: a boolean cannot express "this width did not fit", and the degenerate case — fail at S=1, record 1, S < 1 never true — recovering the old permanent behaviour as a special case rather than a branch is the detail that makes it reviewable.

It cannot merge as it stands: the branch carries 16 compiled objects.

c/COLI_V4_UNIT_ATTENTION_CACHE.o   c/COLI_V4_UNIT_CONFIG.o
c/COLI_V4_UNIT_EXPERT.o            c/COLI_V4_UNIT_EXPERT_STORE.o
c/COLI_V4_UNIT_KV_CACHE.o          c/COLI_V4_UNIT_LAYER.o
c/COLI_V4_UNIT_MATH.o              c/COLI_V4_UNIT_NATIVE_QUANT.o
c/COLI_V4_UNIT_PROMPT.o            c/COLI_V4_UNIT_RESOURCE_PLAN.o
c/COLI_V4_UNIT_SPARSE_ATTENTION.o  c/COLI_V4_UNIT_ST.o
c/build/ownership/COLI_V4_UNIT_{CONFIG,NATIVE_QUANT,RUNTIME,ST}.o

+35/-19 across two source files is what this PR is; the other 16 entries are build products.

This is not your mistake. .gitignore lists every engine binary — colibri, glm, olmoe, inkling, kimi_k3 — and not the one added last. So make deepseek-v4 leaves 26 untracked files sitting in git status, and any git add -A sweeps them in. #800 has the identical 16. #868 adds the missing entries, which stops it recurring for everyone.

If you rebase after #868 lands they will fall out of git status on their own; until then git rm --cached c/COLI_V4_UNIT_*.o c/build/ownership/*.o is enough. Ping me and I will merge it.

On the verification limit you named: agreed, and thank you for naming it rather than implying coverage you did not have. @ThefloorMiner's repro is the one that settles it.

[JustVugg]

Follow-up: #868 is on dev now, so a rebase clears this by itself.

.gitignore gained c/deepseek_v4, c/deepseek_v4.exe and c/COLI_V4_UNIT_*.o, which is where all 16 of those objects came from. After you rebase they will not reappear in git status, and nothing else about the branch needs to change.

Worth saying plainly, because my earlier comment made it sound like your mistake and it was not: the same thing happened to me, on this repo, about ten minutes after I wrote that. I ran make check on the cap branch, did git add -A, and staged the identical 16 files. Caught it in git status before committing purely by chance. That is not a discipline problem on your side, it is a gap in the repo that has now been closed.

Once it is rebased I will merge it. The change is right and I have said so on the substance: recording S instead of a boolean, with a genuine device fault falling out as S < 1 never being true rather than as a separate branch, is the part that makes it reviewable.

@ThefloorMiner's repro is still the only thing that can confirm the runtime behaviour, and your PR says so — which is the right way round.

[ZacharyZcR]

Rebased and clean — git diff --name-only dev...HEAD is c/colibri.c, nothing else. Ready when you are.

One correction, because it will bite the next person and it nearly bit me: the rebase alone did not clear them. Your first comment had it right and the follow-up was too optimistic.

.gitignore has no effect on a path git is already tracking. Those 16 objects were committed, so #868's new rules made them invisible in git status while leaving them in the tree — git ls-tree still listed all 16 after rebasing onto current dev. What removed them was the git rm --cached you named first:

git rm --cached c/COLI_V4_UNIT_*.o
git rm --cached -r c/build/ownership
git commit --amend --no-edit

Same trap as #835, where c/.gitignore already named tests/bench_omp_grain and the binary sat tracked anyway. Ignore rules prevent, they do not remove.

#868 covers 12 of the 16

$ git check-ignore -v c/COLI_V4_UNIT_CONFIG.o
.gitignore:28:c/COLI_V4_UNIT_*.o

$ git check-ignore -v c/build/ownership/COLI_V4_UNIT_ST.o
(no match)

c/build/ is not ignored by anything, so the four objects under c/build/ownership/ still show as untracked after make check and are still one git add -A away from a repeat. My working tree shows ?? c/build/ on current dev right now.

Happy to send the one-line follow-up (c/build/) unless you would rather fold it into #868 yourself.

And it was my discipline problem too

Thank you for saying it happened to you as well, but I have no excuse here: I used git add -A after make test-c on this branch and on #800, and on #850 the same day I staged the file by name and it came out clean. The repo gap is real and now closed; the habit was mine.

#800 is rebased and cleaned the same way. It also picked up 16 new variables that landed in dev since yesterday — mostly #839's MTP/DSpark knobs — which is the registry doing its job again rather than a problem with the branch.