A low-cost, model-aware AMD inference fabric for trillion-parameter sparse MoE models.
ExpertMesh asks a different hardware question: instead of buying one enormous HBM supernode or streaming terabytes of weights through CXL, can a sparse model keep each quantized expert permanently resident on inexpensive HBM and move only compact activations?
The current model-targeted path is Kimi K2 Thinking, an official 1T open-weight MoE target. It is deliberately marked targeted, not compatibility-certified until the real checkpoint and target ROCm hardware pass the documented release gates.
Phase 1 explores that design for configurable open-weight sparse MoE models. Model size, expert count, top-k routing, weight format, and active-weight traffic are explicit profile inputs rather than assumptions embedded in the runtime.
Conventional serving designs optimize a general-purpose GPU cluster. ExpertMesh deliberately co-designs the runtime and hardware around sparse MoE execution:
- Stationary experts: quantized expert weights stay on one owning GPU throughout decode.
- Activation-only transport: RoCE carries routed activations and partial outputs, not model weights.
- Commodity scale-out: refurbished PCIe HBM cards and used EPYC nodes replace a proprietary 64-GPU scale-up domain.
- Topology-aware scheduling: work is grouped by expert owner, with local Infinity Fabric preferred over remote transport.
- Measured hot-expert replication: scarce HBM is spent only after routing traces demonstrate skew.
- Speculative decoding: a smaller draft model can hide network and kernel latency while the large model verifies batches of tokens.
The defensible engineering work is the data plane: checkpoint ingestion, packed-weight HIP kernels, activation dispatch, expert placement, reduction, and evidence-driven scheduling.
| Target | Reference value |
|---|---|
| Compute nodes | 8 refurbished four-GPU EPYC systems |
| Accelerators | 32 × AMD Instinct MI210 64 GB |
| Aggregate HBM | 2 TiB |
| Aggregate HBM bandwidth | 51.2 TB/s theoretical |
| Inter-node fabric | Dual 100/200 GbE RoCE per node |
| GPU board power | 9.6 kW |
| Estimated acquisition cost | $135k–$235k |
| Active-weight planning range | 30–50 GB/token |
| Bandwidth roofline | 1,024–1,707 aggregate token/s |
The throughput range is an ideal bandwidth roofline—not a benchmark or performance promise. It excludes communication, routing imbalance, KDA/attention work, KV traffic, dequantization, kernel inefficiency, and synchronization. See the hardware and cost model and source assumptions.
The checked-in Phase 1 profile uses the publicly announced characteristics of Kimi K3 as one demanding example: approximately 2.8T parameters, 896 experts, top-16 routing, and packed four-bit weights. Those values demonstrate the planner at trillion-parameter scale; they do not make the architecture model-specific. Official checkpoint metadata and licensing always override announcement-era assumptions.
- A HIP routed-expert correctness kernel for group-scaled signed INT4 weights.
- A stable C ABI and documented
[expert, output, packed_input]layout. - A bit-exact NumPy quantizer, dequantizer, routed expert, and sparse MoE reference.
- Deterministic 896-expert placement across 32 stationary owners.
- Activation grouping that preserves token and top-k slot identity.
- Capacity planning and explicitly labeled bandwidth-roofline CLIs.
- A versioned, checksummed checkpoint format and deterministic expert sharder.
- A runnable attention → route → dispatch → INT4 SwiGLU → reduce reference runtime.
- Local and framed-TCP worker transports with numerical parity tests.
- An OpenAI-compatible chat-completions endpoint and three-container deployment example.
- A synthetic GPU smoke test plus eighteen CPU unit/integration tests.
- Architecture, transport, BOM, release gates, sources, and a four-phase roadmap.
flowchart LR
request["Requests"] --> scheduler["Continuous batching and speculation"]
scheduler --> router["Attention, KDA and top-k router"]
router --> dispatcher["Group activations by expert owner"]
dispatcher --> local["Local xGMI hive"]
dispatcher --> roce["Remote RoCE fabric"]
local --> workers["Stationary packed-INT4 experts"]
roce --> workers
workers --> reducer["Weighted top-k reduction"]
reducer --> router
An editable, presentation-ready version is available in FigJam, with repository-native Mermaid and SVG sources.
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest -q
expertmesh-plan configs/phase1-mi210.json
expertmesh-roofline configs/phase1-mi210.jsonOn Windows, activate with .venv\Scripts\Activate.ps1.
Expected planning output begins with:
32 accelerators; 896 experts; minimum free HBM 17.43 GiB/rank
theoretical aggregate decode roofline: 1024–1707 token/s
expertmesh-checkpoint .demo --workers 2
expertmesh-serve .demo --workers 2This creates a tiny deterministic MoE, validates its manifest, assigns experts to two logical
workers, runs the full coordinator/worker data path, and exposes POST /v1/chat/completions on port
8000. For separate worker processes or Docker Compose, see the runtime guide.
The generated model is a plumbing fixture—not a useful language model.
The GPU path requires Linux, ROCm, CMake, and a supported AMD accelerator:
cmake -S . -B build -DEXPERTMESH_BUILD_HIP=ON
cmake --build build -j
./build/expertmesh_hip_smokeThe current kernel prioritizes an auditable layout and correctness. It unpacks signed INT4 weights, applies FP16 group scales, executes the routed expert product, and performs a portable workgroup reduction. The post-checkpoint optimization path replaces this with grouped MFMA GEMM and fused gate/up/down projections. See the kernel contract.
include/expertmesh/ stable HIP runtime ABI
kernels/hip/ packed INT4 routed-expert kernel and GPU smoke test
src/expertmesh/ checkpoint, runtime, transport, serving, planning and reference kernels
configs/ explicit model and hardware assumption profiles
tests/ synthetic correctness and planning tests
docs/ architecture, transport, BOM, gates, evidence and roadmap
| Check | Status |
|---|---|
| Python reference tests | 18 passing |
| Ruff static checks | Passing |
| Package build | Passing |
| Planner and roofline CLI | Passing |
| GitHub Actions | Passing |
| HIP source and smoke test | Included; requires a ROCm GPU runner |
| Target checkpoint compatibility | Requires an immutable checkpoint and reference runtime |
| Performance claims | Intentionally withheld until measured |
This repository does not claim compatibility with any specific trillion-parameter model or that the theoretical roofline will be achieved. Every target—including example targets such as Kimi K3—must provide immutable weights, a compatible license, tensor shapes, quantization semantics, router behavior, attention operators, residual behavior, and bit-exact reference outputs. The mandatory process is documented in release gates.
The architecture is intended for sparse MoE inference. A dense 2.8T model activates nearly all weights per token and has fundamentally different bandwidth economics.
- Architecture
- Hardware and estimated cost
- HIP kernel contract
- Activation transport
- End-to-end reference runtime
- Kimi K2 target profile and bring-up
- Checkpoint release gates
- Roadmap
- Sources and assumptions
- Changelog
Contributions are welcome, especially reproducible AMD benchmarks, grouped MFMA kernels, ROCm transport experiments, and checkpoint-layout adapters. Read CONTRIBUTING.md before opening a pull request. Please report vulnerabilities through the process in SECURITY.md, not a public issue.
Apache-2.0. See LICENSE.