Granite hybrid server startup regression — Spec V2 / mamba-no_buffer contradiction at memory_pool.py:513

[KHAEntertainment]

Symptom

Granite hybrid server fails to start with the following error at python/sglang/srt/mem_cache/memory_pool.py:513:

Spec V2 / mamba-no_buffer contradiction

Workaround: setting SGLANG_ENABLE_SPEC_V2=0 in the environment before launching the server allows it to start cleanly. The workaround is known and works reliably.

Surprising observation

The regression appears to be non-deterministic or environment-sensitive. PR #61 (just merged at the time of this report) was smoke-tested with the granite server using identical launch arguments without the SGLANG_ENABLE_SPEC_V2=0 workaround, and the server started cleanly.

The diff between the PR #61 merge commit (e93e744) and the branch where the regression surfaced (d73b579, PR #52's branch) shows zero changes to memory_pool.py or surrounding initialization code.

Hypotheses to investigate

1. Non-deterministic initialization ordering — the Spec V2 / mamba-no_buffer interaction triggers only when init paths run in a specific order, and minor branch differences elsewhere (or external environment differences) shift the order enough to surface or hide the contradiction.
2. Stale environment variable carry-over — SGLANG_ENABLE_SPEC_V2 may have been set implicitly in the prior session's shell environment and was inherited by the PR fix(ci): validate-sync.sh launches server with every_turn snapshot policy (KHA-352) #61 smoke test process, masking the contradiction. Subsequent fresh shells then surface the issue.
3. Indirect cross-branch difference — something not in memory_pool.py itself but in a sibling file (configs, server_args, model_runner) influences the init path's behavior. Worth grep'ing for any spec_v2 or mamba references in the actual diff between e93e744 and d73b579.

Reproduction

# On the d73b579af branch (or any post-PR-61 main checkout), without env workaround:
python -m sglang.launch_server \
    --model-path /workspace/models/granite-4.0-h-tiny \
    --port 30000 \
    --host 127.0.0.1
# Expected: fails at memory_pool.py:513 with Spec V2 / mamba-no_buffer contradiction

# With workaround:
SGLANG_ENABLE_SPEC_V2=0 python -m sglang.launch_server \
    --model-path /workspace/models/granite-4.0-h-tiny \
    --port 30000 \
    --host 127.0.0.1
# Expected: starts cleanly

Acceptance criteria for the fix

1. The regression is reproduced deterministically on a fresh shell with no inherited environment variables.
2. Root cause is identified (init ordering, stale env, or indirect cross-branch difference).
3. Either (a) a code fix lands so the server starts cleanly without the env workaround, or (b) the contradiction is documented as expected behavior with the env var becoming a documented configuration knob rather than a workaround.
4. Granite hybrid + Codestral pure-Mamba2 + Nemotron-3-Nano-Omni all start cleanly on a single H100 SXM with default arguments after the fix.

Logs and artifacts

Full H100 cascade smoke test logs at s3://engram/test-results/2026-05-03-h100-weekend-track-b/cascade/52/. The CHECKPOINT.md at /workspace/Engram/CHECKPOINT.md (during the H100 session) captured the state at escalation.