Clarify misleading KV cache Storage Throughput metric

[jay-tau]

Problem

The KV cache benchmark uses Storage Throughput for storage_throughput_tokens_per_sec, calculated as:

total_tokens_generated / total_storage_io_latency

That name is confusing because elsewhere storage clearly refers to the disk/NVMe tier, e.g. tier_storage_kv_bytes_read_gb, tier_storage_kv_bytes_written_gb, tier_storage_read_bandwidth_gbps, and tier_storage_write_bandwidth_gbps.

As a result, readers naturally interpret Storage Throughput as raw disk/NVMe throughput, but it is really token throughput through the cache I/O path. CPU RAM hits can increase this metric even while raw disk pressure drops, which makes the discovery note hard to understand:

Storage Throughput shows only 1.1x at cpu_mem=0GB but 2.2x at cpu_mem=4GB

Proposed fix

Rename the displayed/JSON metric to something like:

cache_io_tokens_per_sec

or, if keeping backwards compatibility, add the new name as an alias and mark storage_throughput_tokens_per_sec deprecated in docs/output.

Reserve Storage Throughput wording for actual disk-tier metrics:

tier_storage_read_bandwidth_gbps
tier_storage_write_bandwidth_gbps
tier_storage_kv_bytes_read_gb
tier_storage_kv_bytes_written_gb

Why this matters

The current name makes it look like increasing cpu_mem makes the disk faster. What is actually happening is that CPU RAM reduces cache I/O latency, so tokens / cache_io_latency increases. Raw disk saturation should be judged from tier storage bandwidth/bytes or iostat, not this token metric.

[hazemawadalla]

hey Joel, I think the storage throughput tokens per sec is a valid comparison metric. its total toks processed by storage tier / total storage io time. A higher numerator or a lower denominator will increase this value and the storage tok/s should give you an idea on the mean entry size with the storage KV cache BW metrics.

Drive 1:

"summary": {
"total_requests": 914,
"total_tokens": 168827,
"elapsed_time": 300.0004549026489,
"avg_throughput_tokens_per_sec": 562.7558133362994,
"total_storage_io_time": 29144.53063242731,
"storage_throughput_tokens_per_sec": 5.792750692377138,
"requests_per_second": 3.0466620468845482,
"end_to_end_latency_ms": {
"mean": 147359.97282913723,
"p50": 141980.99168550107,
"p95": 298202.26569068036,
"p99": 322737.99994589004,
"p999": 333787.61627292394,
"p9999": 336579.49411215953
},

Drive 2

"multi_turn_cache_hits": 449,
"multi_turn_cache_misses": 443,
"seed": 42,
"summary": {
    "total_requests": 951,
    "total_tokens": 175854,
    "elapsed_time": 300.12672662734985,
    "avg_throughput_tokens_per_sec": 585.9324891726415,
    "total_storage_io_time": 26909.629612902412,
    "storage_throughput_tokens_per_sec": 6.5349840384158595,
    "requests_per_second": 3.1686614873882997,
    "end_to_end_latency_ms": {
        "mean": 143648.69741565138,
        "p50": 144950.5788530223,
        "p95": 286132.00293149566,
        "p99": 297515.18656505505,
        "p999": 304524.81470455613,
        "p9999": 315042.3143295327
    },
    "storage_io_latency_ms": {
        "mean": 28296.14049726857,
        "p50": 13266.512015950866,
        "p95": 106447.53920135554,
        "p99": 176093.5959397466,
        "p999": 218468.11034832662,
        "p9999": 239120.44118214
    },
    "generation_latency_ms": {
        "mean": 0.0,
        "p50": 0.0,
        "p95": 0.0,
        "p99": 0.0,
        "p999": 0.0,
        "p9999": 0.0
    },

[hazemawadalla]

@dslik @jay-tau

[jay-tau]

Thanks Hazem - agreed this is a useful comparison metric. I’m not arguing to remove it.

The issue is naming. In the current implementation, storage_throughput_tokens_per_sec is:

total_tokens_generated / total_storage_io_latency

But total_storage_io_latency is accumulated from cache I/O across whichever tier served the request, not only the disk/NVMe storage tier. Meanwhile the disk tier is also reported as tier_storage_*, e.g. tier_storage_read_bandwidth_gbps and tier_storage_kv_bytes_read_gb.

That creates a naming inconsistency:

storage_throughput_tokens_per_sec     = cache-wide I/O token throughput
tier_storage_read_bandwidth_gbps      = disk/NVMe read bandwidth
tier_storage_kv_bytes_read_gb         = disk/NVMe bytes read

So storage means “all cache I/O” in one metric, but “disk/NVMe tier” in the adjacent tier metrics. That’s the confusing part.

So “Storage Throughput” reads like raw disk/NVMe throughput, but the metric is really closer to:

cache_io_tokens_per_sec

I’m fine keeping the existing JSON key for compatibility, but I think the displayed name/docs should make this distinction clear. Otherwise cpu_mem=4GB looks like it made the disk faster, when it actually reduced cache I/O latency via RAM hits.