It would be very useful to see the KV Cache Block hit rate improvement that SemBlend introduces. This feels like the metric that then leads to improvements in (for example) TTFT.
Doing this for a known corpus would help potential users evaluate the benefits of semblend over existing hash-exact match KV Cache block stacks.
It would be very useful to see the KV Cache Block hit rate improvement that SemBlend introduces. This feels like the metric that then leads to improvements in (for example) TTFT.
Doing this for a known corpus would help potential users evaluate the benefits of semblend over existing hash-exact match KV Cache block stacks.