Skip to content

cuda: add paged ragged KV research runtime - #795

Open
ZacharyZcR wants to merge 2 commits into
JustVugg:devfrom
ZacharyZcR:feat/paged-ragged-kv-research-runtime
Open

cuda: add paged ragged KV research runtime#795
ZacharyZcR wants to merge 2 commits into
JustVugg:devfrom
ZacharyZcR:feat/paged-ragged-kv-research-runtime

Conversation

@ZacharyZcR

Copy link
Copy Markdown
Contributor

Summary

  • replace grow-copy ragged CUDA KV buffers with fixed 64-token physical pages
  • extend ragged attention and append kernels to consume page tables across page boundaries
  • add per-device CUDA group telemetry and accurate asynchronous GPU timing
  • expose opt-in placement/MTP research controls used by the held-out experiment matrix
  • strengthen cross-page and length-skew ragged-attention coverage

Measured behavior

The 64-token page size was the practical allocation/fragmentation Pareto point. The prototype remains numerically exact across page boundaries and reserves 76.4% less KV memory than fixed-slot allocation on the tested length-skew trace. This PR does not claim a decode throughput gain.

Validation

  • make -C c check: native suite passed; 292 Python tests passed, 18 skipped
  • CUDA compilation is delegated to CI because this workstation has a CUDA GPU but no local nvcc toolkit

Related: #537, #768

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant