Memory offset mismatch during graph replay on A100 (Ampere) — is sm_80 supported?

[Esthesia]

Dear foundry Team

I have a question about hardware support of foundry.

Question

Is NVIDIA Ampere (sm_80, e.g. A100) a supported target for Foundry's graph capture/replay?
The published examples are all on H100/H200 (sm_90), and on my A100 box libcuda_hook.so
aborts replay with a memory-offset mismatch:

[HOOK] ERROR: Memory offset mismatch during replay
[HOOK]   Current offset: 0x60089a800000 (35240.00 MB from base)
[HOOK]   Expected start: 0x60089a600000 (35238.00 MB from base)
[HOOK]   Difference: 2.00 MB

It is properly working on H100 GPU (compute capability 9.0 (Hopper).
The drift is small (2 MB) and reproducible. Happy to gather more diagnostics
if the team can point at what to capture.

Environment

Component	Version
Foundry	0.0.2 (main @ eef1201, "[sglang] Release Foundry 0.0.2")
vLLM	0.22.1.dev0+g0b3ba88f1 (I patched Foundry's modification into vLLM)
PyTorch	2.11.0+cu130
CUDA	13.0
Python	3.11
OS	RHEL 9.6 (kernel 5.14.0-570.12.1.el9_6)
GPU	4× NVIDIA A100-SXM4-40GB, compute capability 8.0 (Ampere)
Driver	590.48.01

For reference, I add patchelf --add-needed libtbb.so.2 on
libcuda_hook.so and the four $ORIGIN symlinks for boost/tbb due to the libtbb.so.2 difference in my library.
The shim loads cleanly (from vllm.compilation.foundry_shim import _AVAILABLE →True) and install_hooks resolves. The failure is at replay time, not load time.

Specific questions

1. Is Ampere (sm_80) intended to work today, or is the hook's
   offset-tracking specific to Hopper's allocator behavior?
2. If supported, is a 2 MB drift indicative of a known issue (alignment,
   workspace, NCCL buffer placement, …) or something I should investigate further on my side?

Thanks!

[xenshinu]

It should support your settings. Did you run two passes on saving?

[Esthesia]

Hi, I ran on two passes on saving.
I run the code vllm recipes.
If it is unusual case, I will check on further and report to you.

[xenshinu]

Can you try our modified pub fork instead of patching the latest vLLM? I haven't test on the new vLLM yet. BTW what model are you running on? are you running on single GPU or multi GPUs? This 2MB is usually the SAVE path deviate from the LOAD path, such as the warmup run before capture is not skipped on the second SAVE.

[Esthesia]

I tested on following the recipe/vllm/README.md.
Python 3.12.13, foundry-org/vllm@foundry (commit 4309c25), wheels.vllm.ai/nightly/cu130 torch nightly, conda Boost 1.85.
I am running on the single GPU case (A100 SXM4 40GB).

SAVE pass 1 OK, SAVE pass 2 OK, LOAD reproduces the bug identically.

[HOOK] ERROR: Memory offset mismatch during replay
[HOOK]   Current offset: 0x60089a800000 (35240.00 MB from base)
[HOOK]   Expected start: 0x60089a600000 (35238.00 MB from base)
[HOOK]   Difference: 2.00 MB

You're right about the SAVE/LOAD path deviation.

• SAVE process: vLLM's pre-capture warmup CUDAGraphs reach register_graph long before prepare_graph_capture. The first warmup's capture_begin → state->register_graph → at::empty → 2 MB grow lands at cursor ~8 GB, well before start_hook_record() is called for any captured graph. The grow sits in the unattributed pre-graph region. By the time the real decode graph_0 capture begins, registered_graphs_ is non-empty and the int64 small pool is already grown — at::empty reuses, delta = 0.

• LOAD process: vLLM does not capture any CUDAGraphs in LOAD mode, so there are zero capture_begin calls before finish_one_graph_load_impl. registered_graphs_ is still empty when finish_one_graph_load_impl calls state->register_graph(graph_0) — that's the first register_graph in the process. The 2 MB grow fires inside finish_one_graph_load_impl, between the point where graph_0's recorded start_base_addr (35238 MB) was supposed to equal the live cursor and replay_hook_events_from_json's start check. Cursor advances 2 MB unaccounted, check fails with current=35240 vs expected=35238, replay aborts.

[xenshinu]

The first SAVE runs warmup before capture but the second SAVE does not, and thus all transient memory allocation during graph capture will be captured by Foundry, and Foundry replays them during the finish_one_graph_load. The error at graph_0 means the memory layout has deviated before the start of memory restoration.

Can you confirm the error happens on our example recipe running Qwen3-1.7b on a single A100?
Can you share me the full log of second SAVE and LOAD?

You may also uncomment this line

foundry/csrc/hook.cpp

Line 7 in eef1201

// #define HOOK_DEBUG

to show debug message.

We will improve the debugging output in future releases.

[Esthesia]

Thank you for the prompt answering.
I set the HOOK_DEBUG option and this is the log files.

save1.log
save2.log
load.log

Please let me know if I set something wrong. Hope this helps you find out the reason.

[xenshinu]

Hi @Esthesia, I've basically located the bug comes from different workspace size of flash attention 2. Sorry about the delay and thanks for pointing it out.
Please try #4

[xenshinu]

Close as completed.