General Questions on Sim

[utkmohan]

Hi!!

First of all, great work on LLMServing Sim. I had some questions on the sim, would be appreciated if you can help me out in understanding.

1. Looking at your given trace for certain GPUs/accelerator in /llm_profile/perf_models/A6000/meta-llama/Llama-3.1-8B/tp1/layers.csv
   it seems for per input length, latencies of 15 layers are collected on real hardware.

I tried to re-run and collect traces it seems some layers are missing when input length is high? Also the layer_name "attn" is also missing in the 15 layers being collected. Maybe that can be fixed by adding self._attn_timer = Timer(name="attn") in /llm_profile/models/llama.py around L:241 and wrapping L:275-284 ?

2. I see that you just corrected the return parser.parse_args() in /layers/main.py (I was wondering why I wasn't able to capture latencies!!)

3. with prefix cache sharing enabled it seems the prefix cache is eagerly being promoted to cache, even if there is no future re-use of the prefix? shouldn't it be cached only if there is a future req that comes in with same prefix (input tokens or input+decoded output tokens)? just wondering if that takes up a lot of cache space!!

4. what exactly is the purpose of --enable-local-offloading? it seems when you define the NPU config, model weights are stored in NPU anyways? why offload weights on NPU if they are already on NPU?

I'd appreciate your thoughts and inputs on this.

[JaehongCS20]

Thanks for your interest and for the detailed questions.

1. It is fine that attn is not measured in that trace. In our framework, attention is modeled separately from the per-layer measurements. We are also currently overhauling the llm_profiler, so if you are following the updates, you will likely see related changes soon.
2. Yes, there was indeed a small but critical bug there. 😅
3. Regarding the prefix cache sharing behavior, that part was implemented quite a while ago, so I need to look into it again before I can give you a precise answer.
4. By default, we assume all weights are already resident on the NPU, so in the LOCAL case no weight loading is modeled. If --enable-local-offloading is enabled, we instead assume the weights start on the CPU, and each layer fetches its weights from CPU before execution.

[utkmohan]

Thanks for your response. By any chance do you have a timeline on the changes of llm_profiler ? I'd really like to try out some stuff out but it seems currently the traces are incomplete for certain gpus and they tps!! Current script doesnt accurately capture the exact same per layer latency, I understand attention is being taken from attention.csv but my runs seem to be missing and also inaccurate?

May I ask the traces you have provided what profiling method did you use in /layers/main.py??

[Edit]: For the 8B model it seems that I am able to get the traces, its when I go for larger models like 70B is when the default profiling method i.e --profile-method record-function fails [it isn't able to capture certain layer latencies]!!

[JaehongCS20]

Hi, there’s a new vLLM-based profiler available now. (renamed from llm_profiler to profiler)

Please give it a try.

It may take a bit longer because of the attention skew step.

[utkmohan]

Hi,

Thanks for the update. I profiled a new Llama model which took some time but I understand that you moved to vLLM profiler.
Quick question, the entire simulator seems to have become much slower end to end to simulate any given config compared to previous version. Previous version would complete simulations within 5-10 min it seems this update makes the sim run slow as well? is there any reason behind that?

[JaehongCS20]

No, the logic itself is the same as before, so it shouldn’t take that long. What configuration did you use?

[utkmohan]

I'm running Llama 3.1 70B model profiled on A100 gpu. for a similar configuration of 8 instances (colocated PD) I get different results in the prev sim vs the new updated version of sim for same dataset.

Does vLLM profiler give more accurate latency profiling? I didn't dig too deep into that. For the same dataset and config new sim gives better throughput and lower TTFT? could it be because of the profiling method change?

[JaehongCS20]

8 instances could take a while, depending on the configuration.

vLLM profiling is much more accurate. I’m not sure which commit you’re referring to as new-sim, but one possible reason for the improved throughput and lower TTFT is that prefix caching is enabled by default now.

Also, in the latest commit, we added a hotfix for a prefix caching issue. That fix would generally make the simulation slower rather than faster, so it likely doesn’t explain the speedup.

[utkmohan]

Sorry I should have been more clear, by new-sim I meant the version which is on main currently and the previous one is the v1.0 I believe in your branch now.

I do see that prefix caching is enabled but my trial runs were by enabling prefix caching so I don't suppose thats the issue.

I also see that you incorporated the reqs being sorted in arrival time order and then being routed. I actually had done something similar while local development to try something out.

Let me work on some cases and get back to you since I want to try out couple of things.

Quick question, there's no inter-worker/instance communication right? If I were to implement that what would you recommend doing or changes in backend? I see for P/D disaggregation you included the communication time by including an extra npu as prefill worker right? but this sim yet doesnt fully support inter-instance communication I believe. Let me know your thoughts!!

Edit: By inter-instance I meant within a node. I see thats there a link_bw and latency for inter-node communication.