herdsman-npu-runtime

Rust NPU GenAI runtime service for Herdsman.

The service exposes an OpenAI-compatible text-generation API and a Herdsman single-model preload API. Vendor SDKs are loaded only by worker processes.

Quick Start

Build (Windows + MSVC)

# One-step build: downloads dependencies, compiles C++/Rust, and stages to dist/
.\scripts\build-all.ps1

Large runtime DLL bundles are not stored in Git. During staging, the build script downloads the published Intel/AMD runtime packages from ModelScope into dist-packages/ and expands them into dist/herdsman-npu-runtime/.

For detailed build instructions, troubleshooting, and advanced options, see BUILD.md.

Run

cd dist\herdsman-npu-runtime
.\npu-server.exe --config ..\..\configs\models.example.json --port 8099

Or from source (development):

cargo run -p npu-server -- --config configs/models.example.json --port 8099

Smoke Test

# Health check
Invoke-RestMethod http://127.0.0.1:8099/health

# Preload a fake model
Invoke-RestMethod -Method Post http://127.0.0.1:8099/v1/herdsman/preload `
  -ContentType application/json `
  -Body '{"model":"fake-npu-chat","provider":"fake","vendor":"intel"}'

Project Structure

herdsman-npu-runtime/
├── crates/
│   ├── npu-server/          # HTTP server (Axum)
│   ├── npu-worker/          # Worker process (loads vendor SDKs)
│   ├── npu-worker-protocol/ # IPC protocol
│   ├── npu-api/             # Shared API types
│   ├── provider-intel-ov/   # Intel OpenVINO GenAI provider (FFI)
│   ├── provider-amd-oga/    # AMD ONNX Runtime GenAI provider
│   └── provider-fake/       # Mock provider for testing
├── native/
│   └── intel-ovgenai-shim/  # C++ wrapper for OpenVINO GenAI
├── scripts/
│   ├── build-all.ps1        # Complete build pipeline
│   ├── build.ps1            # Main build script
│   └── setup-deps.ps1       # Download OpenVINO SDK
├── tools/
│   └── stage-runtime.ps1    # Package dist/ directory
└── configs/
    └── models.example.json  # Example model catalog

Architecture

• npu-server: Axum HTTP server, manages worker processes
• npu-worker: Isolated process that loads vendor SDKs (Intel/AMD)
• Providers: Plugin system for NPU vendors
  • provider-intel-ov: Intel OpenVINO GenAI via C++ shim DLL
  • provider-amd-oga: AMD ONNX Runtime GenAI (WIP)
  • provider-fake: Test mock
• C++ Shim: Prevents symbol conflicts between server and workers

Development

Prerequisites

• Rust toolchain (x86_64-pc-windows-msvc)
• Visual Studio 2022 with C++ workload
• CMake 3.22+ (bundled with VS or standalone)

Build Components Separately

# C++ shim only
.\scripts\build.ps1 -SkipRust -SkipStage

# Rust only
.\scripts\build.ps1 -SkipNative -SkipStage

# Skip dependency download (if already present)
.\scripts\build.ps1 -SkipDeps

Clean Build

.\scripts\build-all.ps1 -Clean

Integration with Herdsman

This runtime is designed as a standalone service for the main Herdsman application:

1. Herdsman detects Intel/AMD NPU hardware
2. Launches npu-server.exe as a child process
3. Communicates via REST API:
   • /v1/herdsman/preload - Load model into worker
   • /v1/chat/completions - OpenAI-compatible inference
   • /health - Health check

Environment variables:

• HERDSMAN_NPU_RUNTIME_EXE - Path to npu-server.exe
• HERDSMAN_NPU_RUNTIME_CONFIG - Path to model catalog JSON

License

MIT