Skip to content

inkling: ring-buffer KV cache for sliding-window layers (~10x less KV memory at long context) - #830

Open
dpanelli wants to merge 2 commits into
JustVugg:devfrom
dpanelli:swa-kv-ring-buffer
Open

inkling: ring-buffer KV cache for sliding-window layers (~10x less KV memory at long context)#830
dpanelli wants to merge 2 commits into
JustVugg:devfrom
dpanelli:swa-kv-ring-buffer

inkling: make kv_alloc grow-with-copy ring-aware (#786 collision)

bccdcb5
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs