interweave: publish native CUDA server, add CI, document the offload path#31
Merged
Conversation
Adds the production multi-stream CUDA matmul server (gpu_server_ms.cu) that serves the POWER8 to C4130 offload, with build and run docs. It was only on the deployment box until now. Adds a CI workflow running the CPU-runnable matmul server test suite on Python 3.10 and 3.12. First real CI for this repo. Documents the matmul offload path in the interweave README: the Python reference server, the native production server, and how to run the tests. Signed-off-by: Scott <scottbphone12@gmail.com>
pytest was importing exo/__init__.py via package discovery, which needs the full runtime deps (psutil and friends). The suite is self-contained, so copy the three files to a temp dir and run there. Signed-off-by: Scott <scottbphone12@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #30. Three things to make this repo actually useful to someone who is not me:
Publishes the native multi-stream CUDA matmul server (gpu_server_ms.cu) that has been serving the POWER8 to C4130 offload in production. Per-connection cuBLAS lanes, refcounted Q4_K weight cache, wire protocol v4. Until now it only existed on the deployment box. Build verified with CUDA 12.0.
Adds a CI workflow that runs the CPU-runnable matmul server test suite (14 tests) on Python 3.10 and 3.12. First real CI on this repo. Verified the suite passes with no GPU visible.
Documents the whole matmul offload path in the interweave README: reference Python server, native production server, ports, and how to run the tests.