Question: Where is the KV Cache Transfer delay formula implemented in the source code?

[MskSg-tech]

Hi LLMServingSim team,

Thank you so much for developing such a fantastic simulation framework and providing great documentation!

Question

In the official documentation for the Disaggregated (Prefill-Decode Split) example, it states:

"The cache transfer shows up as a one-time cost between prefill end and first decode step (modeled via link_bw)."

I am running experiments with this split setup and want to inspect the underlying source code to see how this transfer delay is implemented.

Could you please point me to the exact file and function/line of code where this KV cache transfer cost is actually calculated and applied to the simulation ticks?

Is it computed on the Python side before passing to the backend simulator, or is it handled within the C++ network backend?

Thank you so much for your help!

[JaehongCS20]

Hi, thanks for the question!

In the current flow, the KV cache transfer size is attached on the Python side to the generated .txt trace file. Then, the Chakra graph converter reads that information and converts the KV cache transfer into communication nodes in the execution graph.

After that, the generated execution graph is passed to ASTRA-sim, and the actual transfer is handled by ASTRA-sim’s network backend according to the configured network parameters such as link_bw.

So, the transfer is not directly computed as a Python-side tick delay before being passed to the backend. Instead, Python provides the KV cache transfer size in the trace, Chakra converts it into the execution graph, and ASTRA-sim’s network backend simulates the communication cost.

The main code path that models this transfer in the converter is in Chakra here:

https://github.com/casys-kaist/chakra/blob/main/src/converter/llm_converter.py#L734-L757

Hope this helps clarify the implementation path!

[MskSg-tech]

Hi, thank you so much for the quick and incredibly clear explanation!

I checked the llm_converter.py source code you pointed out, and now I perfectly understand how the KV cache transfer is converted into communication nodes via Chakra.

Let me ask the following question.

Is this the exact code path in ASTRA-sim?

In my current Analytical configuration, is this KV transfer node processed, and is its serialization delay calculated inside congestion_unaware/basic-topology/BasicTopology.cpp (specifically within the compute_communication_delay function shown below)?

auto serialization_delay = static_cast<double>(chunk_size) / bandwidth_Bpns;
auto comms_delay = link_delay + serialization_delay;

[DevHSA]

"In the current flow, the KV cache transfer size is attached on the Python side to the generated .txt trace file." It would be great if you can point me to the location of this generated .txt trace file.

The use case here is to model a DPU accelerated disaggregated KV cache architecture with emphasis on the decode phase. Put simply, I want to simulate a scenario for the decode phase where the KV Cache is offloaded to a network attached storage (basically fetching KV cache adds extra latency -- either on rack or off rack). I understand that this extra latency currently can be simulated indirectly using the KV cache transfer size as mentioned above. Is this a knob that can be tuned? Is there a better way to do this? Any leads would be very helpful.

[JaehongCS20]

For the generated .txt trace file, you can find some example traces under https://github.com/casys-kaist/astra-sim/tree/main/inputs/trace/examples. The actual trace structure can also be inspected by running any example and looking at the generated trace in astra-sim/inputs/trace.

Regarding the DPU-accelerated disaggregated KV cache use case, if the KV cache access goes through the network, then the modeled overhead should be affected by the configured link bandwidth and link latency.

As far as I know, ASTRA-sim does not currently have a dedicated storage device model. However, one possible approximation would be to treat remote memory as a storage-like backend and configure its bandwidth and latency to match the target network-attached storage path, whether on-rack or off-rack.

So I think this can currently be approximated through transfer size plus network/remote-memory bandwidth and latency settings. A more accurate model would likely require adding an explicit storage/DPU path.