perf(dflash): split-chain fast rollback with F32 SSM checkpoints (+29% decode)

[Amidwestnoob]

Summary

• add layer-split F32 SSM checkpoint capture and fast pure-chain rollback for DFLASH
• keep production tree verification fail-closed and reject sibling/non-chain geometry before execution
• add a CPU-only root-inclusive pure-chain guard regression test

Benchmark

Controlled dual-GPU A/B/A benchmark on two RTX 3090 24 GB cards using the same pinned models, prompt, generation length, power/thermal conditions, and layer split (--target-gpus 0,1 --target-layer-split 12,11). Each block used one discarded warmup and five measured runs.

Metric	Baseline	Candidate	Delta
Median decode throughput	22.76 tok/s	29.41 tok/s	+29.22%
Median wall time	24.895 s	22.560 s	-9.38%
Prompt processing	~95.4 tok/s	~95.4 tok/s	flat
Acceptance rate	42.4%	37.2%	-5.2 pp
Average commit	7.76	5.95	-1.81

All 15 measured runs produced byte-identical 256-token output:

1653a412811d3b87e26aab900e801b411fd68f9214b512ae984db6404b82204f

A1/A2 baseline drift was 0.48%. Maximum observed GPU temperature was 64 C.

Memory trade-off

The F32 SSM checkpoint storage adds exactly 1,774,190,592 bytes (~1.65 GiB) of persistent GPU memory in the tested split configuration. Both GPUs are required for the tested 27B configuration; a single RTX 3090 is not sufficient.

Safety boundary

• supports_tree_verify() remains unconditionally false in production.
• Only the validated root-inclusive pure-chain rollback path is enabled.
• Sibling, malformed-root, skipped-parent, cyclic/forward-parent, out-of-range, truncated, and null geometry is rejected before the execution sentinel.
• Fast rollback is opt-in through DFLASH_SPLIT_FAST_ROLLBACK and requires successful capture validation.
• Temporary evidence instrumentation is excluded from this commit.

Verification

• rebased onto current upstream main at 5e302cbb483819cd21e72f5dd8becaa609eca8cf
• preserved upstream fix(qwen35): route non-tree GDN capture through persist buffer (src[7]) #469 non-tree GDN persist-buffer routing through src[7]
• full Release CUDA 8.6 build and link: passed
• current upstream test suite: 15/15 passed, including CUDA and the pure-chain guard
• rebased-candidate dual-GPU validation: passed; 43/43 fast rollbacks succeeded with zero fallback
• rebased validation produced the same generated suffix hash as the controlled benchmark
• controlled A/B/A benchmark: 18/18 isolated processes passed, including warmups
• no OOM, CUDA, fallback, tree-path, output-parity, or thermal failures

Scope

Measured on the exact dual-RTX-3090 configuration above with a 1,024-token prompt and 256 generated tokens. No performance claim is made for single-GPU execution, sibling-tree verification, other models, or other hardware.

[cubic-dev-ai]

All reported issues were addressed across 13 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[davide221]

@Amidwestnoob thanks for the contribution, please review and fix cubic reviews

[Amidwestnoob]

@howard0su When you have a chance, could you please review this? Cubic passes and all four review threads are resolved. The latest follow-up gates capture logging behind the existing self-test/diagnostic flags; the Release CUDA build completed and all 15 tests passed locally.

[dusterbloom]

@Amidwestnoob I did run it and there is still little gain that can be squeezed.

See my PR feel free to diff it and cherry pick the commits I added on top of your good PR 506

[Amidwestnoob]

Added 263cf60 (perf(qwen35): batch recurrent snapshot synchronization), adapted from dusterbloom's bb76966 while preserving the later split-capture diagnostic gate.

Verification on the exact PR head lineage:

• Built dflash_common, test_recurrent_snapshot, test_qwen35_split_tree_guard, test_dflash, test_generate, dflash_server, backend_ipc_daemon, and test_server_unit.
• Focused CTest: recurrent_snapshot, qwen35_split_tree_guard, and server_unit — 3/3 passed.
• Actual single-GPU and dual-GPU layer-split server health/chat flows passed.
• Three measured runs per build after warmup, same RTX 3090/model/workload:
  • snapshot + restore median: 1.79 → 1.12 ms/step (37.43% reduction)
  • decode median: 40.43 → 40.61 tok/s
• All six measured baseline/patched output streams were byte-identical (25ab6ff1b0ce3b3e7ed9bd6a453e198de0ee0df9813630775c30efe228de8183).

The bounded divergence check also found a pre-existing target-vs-DFlash mismatch at generated token 111 in both legacy and fast modes, before their restore-path outputs differ. That finding is not introduced by this batching commit and should be investigated separately in verifier/acceptance/state-selection semantics.

[cubic-dev-ai]

All reported issues were addressed across 9 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>