From 55e8442ddcef4ad8aedc16c80a750ed52305c6a3 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 21:32:33 +0800 Subject: [PATCH 01/12] chore: update readme and enable more build features --- README.md | 495 ++++-------------------------------------------------- 1 file changed, 32 insertions(+), 463 deletions(-) diff --git a/README.md b/README.md index 56972c0..312fefa 100644 --- a/README.md +++ b/README.md @@ -1,500 +1,69 @@ # XLAI -**Documentation site:** the VitePress site lives under [`docs/`](docs/). Build and preview locally with `pnpm docs:dev` and `pnpm docs:build`. +`xlai` is a **Rust-first** AI integration workspace for building reusable AI calling flows across **native** applications and the **browser**, behind a unified API with pluggable backends. The name was inspired by [moeru-ai/xsai](https://github.com/moeru-ai/xsai). -`xlai` is a Rust-first AI integration workspace for building reusable AI calling flows across native applications and the browser. -The name was inspired by [moeru-ai/xsai](https://github.com/moeru-ai/xsai), which focuses on extra-small AI SDK. +> **Status:** early stage; the public API is still evolving. -The project is designed around a unified API with pluggable backends. Today it includes: +## Documentation -The long-term goal is to support API-based models, tool integration, skill management, knowledge retrieval, vector search, and local/device inference behind the same overall API model. +The full documentation lives in the VitePress site under [`docs/`](docs/): -## Status +| Topic | Link | +| ---------------------- | ----------------------------------------------------------- | +| Introduction | [`docs/guide/index.md`](docs/guide/index.md) | +| Install and build | [`docs/guide/getting-started.md`](docs/guide/getting-started.md) | +| Environment variables | [`docs/guide/configuration.md`](docs/guide/configuration.md) | +| Architecture & crates | [`docs/architecture/index.md`](docs/architecture/index.md) | +| Crate taxonomy | [`docs/development/crates-taxonomy.md`](docs/development/crates-taxonomy.md) | +| Rust SDK | [`docs/rust/index.md`](docs/rust/index.md) | +| JavaScript / WASM | [`docs/js/index.md`](docs/js/index.md) | +| Provider support | [`docs/providers/index.md`](docs/providers/index.md) | +| QTS (Qwen3 TTS) | [`docs/qts/index.md`](docs/qts/index.md) | +| CI, testing, releases | [`docs/development/index.md`](docs/development/index.md) | -This repository is in an early stage and the public API is still evolving. +Preview the site locally: -Already implemented: - -- Cargo workspace with native and `wasm32` build support -- OpenAI-compatible chat backend -- chat sessions with per-session tool registration -- streaming chat output -- configurable tool-call execution mode: concurrent by default, sequential optional -- unit/mock test lane and protected e2e test lane -- GitHub Actions for build, test, formatting, clippy, and e2e - -## Workspace Layout - -```text -xlai/ -├── vendor/ -│ └── native/ # git submodules: llama.cpp, ggml (for xlai-sys-*) -├── crates/ -│ ├── core/ -│ │ └── xlai-core/ -│ ├── runtime/ -│ │ └── xlai-runtime/ -│ ├── backends/ -│ │ ├── xlai-backend-openai/ -│ │ ├── xlai-backend-transformersjs/ -│ │ ├── xlai-backend-gemini/ -│ │ └── xlai-backend-llama-cpp/ -│ ├── qts/ -│ │ ├── xlai-qts-core/ -│ │ ├── xlai-qts-manifest/ -│ │ └── xlai-qts-cli/ -│ ├── sys/ -│ │ ├── xlai-build-native/ -│ │ ├── xlai-sys-llama/ -│ │ └── xlai-sys-ggml/ -│ └── platform/ -│ ├── xlai-facade/ -│ ├── xlai-native/ -│ ├── xlai-wasm/ -│ └── xlai-ffi/ -├── packages/ -│ └── xlai/ -└── .github/workflows/ +```bash +pnpm install +pnpm docs:dev ``` -For crate boundaries and request flow, see [ARCHITECTURE.md](ARCHITECTURE.md). For publish vs internal crates, see [docs/development/crates-taxonomy.md](docs/development/crates-taxonomy.md). - -### Crates And Packages - -- `crates/core/xlai-core` - Shared domain types and traits for chat, tools, embeddings, knowledge, and vector search. -- `crates/platform/xlai-ffi` - Native C ABI facade crate for future FFI integrations. -- `crates/runtime/xlai-runtime` - Runtime builder, chat session API, streaming, and tool-calling orchestration. Local-backend prompt prep (`PreparedLocalChatRequest`, tool JSON, templates) lives in **`xlai_runtime::local_common`** (used by llama.cpp and transformers.js backends). -- `crates/platform/xlai-facade` - Internal native aggregate wiring and re-exports for `xlai-native` only (not used by `xlai-wasm`). Not published to crates.io. -- `crates/platform/xlai-native` - Native Rust-facing entrypoint: explicit re-exports, optional `qts`, and `gemini` submodule for workspace-only Gemini types. -- `crates/platform/xlai-wasm` - Browser-facing `wasm-bindgen` crate for web integration. Default Cargo feature `qts` enables local QTS entrypoints (`qtsBrowserTts`, manifest validation); the in-browser engine is still a stub until GGML/ORT WASM work lands (see `docs/qts/wasm-browser-runtime.md`). -- `crates/backends/xlai-backend-llama-cpp` - Native `llama.cpp` chat backend for local GGUF inference. -- `crates/sys/xlai-build-native` - Shared `build.rs` helpers for native CMake/OpenBLAS/Vulkan and llama.cpp CMake patches (build-dependency only). -- `crates/sys/xlai-sys-llama` - Vendored `llama.cpp` native stack for the local chat backend. -- `crates/sys/xlai-sys-ggml` - Vendored standalone `ggml` native stack for QTS. -- `packages/xlai` - Vite-based TypeScript package published as `@yai-xlai/xlai`, built on top of `xlai-wasm`, with Vitest coverage. -- `crates/backends/xlai-backend-openai` - OpenAI-compatible backend implementation using `reqwest`. -- `crates/backends/xlai-backend-transformersjs` - Browser chat backend that delegates generation to a JavaScript adapter (WASM). -- `crates/backends/xlai-backend-gemini` - Google Gemini HTTP backend (workspace-only; `publish = false`). -- `crates/qts/xlai-qts-manifest` - Serde types for browser QTS model manifests and capability JSON (no GGML/ORT); used by `xlai-wasm` (feature `qts`) and re-exported as `xlai_qts_core::browser`. -- `crates/qts/xlai-qts-core` - Qwen3 TTS engine; links standalone `ggml` through `xlai-sys-ggml`. Exposes native `TtsModel` (`QtsTtsModel`, WAV output; tuning via `TtsRequest` metadata `xlai.qts.*`). **`VoiceSpec::Clone`** uses the first reference sample (inline WAV only): x-vector and ICL prompts, with optional `xlai.qts.voice_clone_mode` (`xvector` \| `icl`). ICL needs `qwen3-tts-reference-codec.onnx` + preprocess JSON from `uv run export-model-artifacts` (see `docs/qts/export-and-hf-publish.md`). Pipelined vocoder chunking and overlap are documented in `docs/qts/vocoder-streaming.md`. `xlai_qts_core::browser` re-exports `xlai-qts-manifest`. -- `crates/qts/xlai-qts-cli` - Binary `xlai-qts`: `synthesize`, `profile`, and interactive `tui`. Without voice-clone flags, `synthesize` uses `xlai-runtime` + `xlai-qts-core` (`QtsTtsModel`). With `--voice-clone-prompt` or `--ref-audio`, it uses the direct engine path. Run `cargo run -p xlai-qts-cli -- --help` (or `… synthesize --help`) for flags. - -## Requirements +## Quick start -- Rust stable -- minimum supported Rust version: `1.94` - -The workspace uses: - -- edition `2024` -- Apache 2.0 license - -## Quick Start - -### Build +Requirements: Rust stable (MSRV **1.94**, edition **2024**) and **pnpm**. ```bash +git clone https://github.com/yetanother.ai/xlai.git +cd xlai cargo build --workspace -``` - -### Test - -```bash cargo test --workspace ``` -### Check `wasm32` - -```bash -rustup target add wasm32-unknown-unknown -cargo check -p xlai-wasm --target wasm32-unknown-unknown --features qts -``` - -### Install JavaScript dependencies +JavaScript workspace: ```bash pnpm install -``` - -### Build `@yai-xlai/xlai` - -```bash pnpm --filter @yai-xlai/xlai build -``` - -### Test `@yai-xlai/xlai` - -```bash pnpm --filter @yai-xlai/xlai test ``` -### Run clippy - -```bash -cargo clippy --workspace --all-targets -- -D warnings -``` - -### Check formatting - -```bash -cargo fmt --all -- --check -``` - -## Local Configuration - -For local end-to-end testing, copy the template and fill in your values: - -```bash -cp .env.example .env -``` - -Current variables: - -- `OPENAI_API_KEY` -- `OPENAI_BASE_URL` -- `OPENAI_MODEL` -- `OPENAI_TRANSCRIPTION_MODEL` - -Local `.env` files are ignored by Git. - -## Example - -This is the current native Rust usage style: - -```rust -use xlai_native::core::{ - ToolCallExecutionMode, ToolDefinition, ToolParameter, ToolParameterType, ToolResult, -}; -use xlai_native::{OpenAiConfig, RuntimeBuilder}; - -#[tokio::main] -async fn main() -> Result<(), Box> { - let runtime = RuntimeBuilder::new() - .with_chat_backend(OpenAiConfig::new( - std::env::var("OPENAI_BASE_URL") - .unwrap_or_else(|_| "https://api.openai.com/v1".to_owned()), - std::env::var("OPENAI_API_KEY")?, - std::env::var("OPENAI_MODEL") - .unwrap_or_else(|_| "gpt-4.1-mini".to_owned()), - )) - .build()?; - - let mut chat = runtime - .chat_session() - .with_system_prompt("Be concise."); - - chat.register_tool( - ToolDefinition { - name: "lookup_weather".into(), - description: "Lookup current weather".into(), - parameters: vec![ToolParameter { - name: "city".into(), - description: "City name".into(), - kind: ToolParameterType::String, - required: true, - }], - execution_mode: ToolCallExecutionMode::Concurrent, - }, - |arguments| async move { - let city = arguments["city"].as_str().unwrap_or("unknown"); - Ok(ToolResult { - tool_name: "lookup_weather".into(), - content: format!("weather for {city}: sunny"), - is_error: false, - metadata: Default::default(), - }) - }, - ); - - let response = chat.prompt("What's the weather in Paris?").await?; - println!("{}", response.message.content); - - Ok(()) -} -``` - -Agent sessions are available through the same runtime: - -```rust -use futures_util::StreamExt; -use xlai_native::core::{ - ChatChunk, ToolCallExecutionMode, ToolDefinition, ToolParameter, ToolParameterType, ToolResult, -}; -use xlai_native::{ChatExecutionEvent, OpenAiConfig, RuntimeBuilder}; - -#[tokio::main] -async fn main() -> Result<(), Box> { - let runtime = RuntimeBuilder::new() - .with_chat_backend(OpenAiConfig::new( - std::env::var("OPENAI_BASE_URL") - .unwrap_or_else(|_| "https://api.openai.com/v1".to_owned()), - std::env::var("OPENAI_API_KEY")?, - std::env::var("OPENAI_MODEL") - .unwrap_or_else(|_| "gpt-4.1-mini".to_owned()), - )) - .build()?; - - let mut agent = runtime - .agent_session()? - .with_system_prompt("Use tools when helpful."); - // Streaming tool loop: - // - `.with_max_tool_round_trips(n)` — cap tool round-trips (default 8). - // - `.with_context_compressor(|msgs, est| async move { Ok(msgs) })` — optional Rust-only hook - // before each streamed model call (see README “Context compression hook”). - - agent.register_tool( - ToolDefinition { - name: "lookup_weather".into(), - description: "Lookup current weather".into(), - parameters: vec![ToolParameter { - name: "city".into(), - description: "City name".into(), - kind: ToolParameterType::String, - required: true, - }], - execution_mode: ToolCallExecutionMode::Concurrent, - }, - |arguments| async move { - let city = arguments["city"].as_str().unwrap_or("unknown"); - Ok(ToolResult { - tool_name: "lookup_weather".into(), - content: format!("weather for {city}: sunny"), - is_error: false, - metadata: Default::default(), - }) - }, - ); - - // Unary `prompt` / `execute` / `prompt_*` = one model call (no tool callbacks). - // Use `stream_prompt` / `stream` / `stream_*` for the multi-round tool loop. - let mut stream = agent.stream_prompt("What's the weather in Paris?"); - let mut response = None; - while let Some(item) = stream.next().await { - let item = item?; - if let ChatExecutionEvent::Model(ChatChunk::Finished(resp)) = item { - response = Some(resp); - } - } - let response = response.expect("stream ended without a finished model response"); - println!("{}", response.message.content); - - Ok(()) -} -``` - -Add `futures-util` to your crate’s dependencies (same major line as the workspace) so `StreamExt` is available. - -For local native inference, the same builder can be pointed at `llama.cpp`: - -```rust -use xlai_native::{LlamaCppConfig, RuntimeBuilder}; - -#[tokio::main] -async fn main() -> Result<(), Box> { - let runtime = RuntimeBuilder::new() - .with_chat_backend( - LlamaCppConfig::new(std::env::var("LLAMA_CPP_MODEL")?) - .with_context_size(4096) - .with_max_output_tokens(256), - ) - .build()?; - - let response = runtime - .chat_session() - .with_system_prompt("Be concise.") - .prompt("Reply with a short greeting.") - .await?; - - println!("{}", response.message.content); - Ok(()) -} -``` - -## Tool Calling - -`Chat` sessions can register tools directly with async callbacks. `Agent` sessions -also expose an `McpRegistry` via `agent.mcp_registry()` so MCP-provided tools can -be registered separately from built-in agent tools. - -The WebAssembly package mirrors this split with `chat(...)`, `createChatSession(...)`, -`agent(...)`, and `createAgentSession(...)`. - -Current behavior: - -- tools are registered per chat or agent session -- tool calls are exposed to the model through the runtime request -- **`Chat`** performs one model call per `prompt` / `execute` / `stream`; it does not execute tool callbacks or run multiple model rounds—you get the model’s response (including any `tool_calls`) and can drive the next step yourself -- **`Agent`** - - **Unary** `prompt` / `execute` / `prompt_parts` / `prompt_content`: exactly **one** model call; registered tools are **not** run by the runtime (same “no silent multi-minute loop” guarantee as chat). - - **Streaming** `stream` / `stream_prompt` / `stream_prompt_content` / `stream_prompt_parts`: runs the automatic **tool loop**. After each assistant turn that finishes with tool calls, tools run and another model turn starts until a response has no tool calls or **`Agent::with_max_tool_round_trips`** (default `8`) is exceeded. -- when tools run (on **`Agent`**), local session tools are used before falling back to a runtime-level tool executor -- each tool’s `ToolDefinition::execution_mode` controls how its calls interact with other calls in the same model turn -- if any invoked tool in a turn is `Sequential`, all tool calls in that turn run sequentially in model order (no overlap) -- otherwise, tool calls in a turn run concurrently - -### Tool loop - -| Surface | Control | Effect | -| ---------------- | ---------------------------------- | --------------------------------------------------- | -| **Rust `Agent`** | `with_max_tool_round_trips(usize)` | Maximum model↔tool rounds per stream (default `8`). | - -`Chat` never runs this loop. Unary agent calls never block on long multi-round tool execution without you choosing a streaming API. - -### Context compression hook - -On **`Agent`**, **`with_context_compressor`** (Rust) registers an **async** closure that runs **once per streaming tool-loop round**, immediately **before** each model call. It receives the full accumulated `ChatMessage` list for that stream and a **best-effort** `Option` input-token estimate (JSON serialization of the outgoing `ChatRequest`, bytes÷4 heuristic—not provider-tokenizer-accurate). It must return the message list to send for that round (often a compressed copy). The agent still appends assistant and tool messages to its **internal** history after each round; only the **outgoing** request for that step uses the returned list. Returning an empty list fails the stream with a provider error. **`Agent::register_context_compressor`** is the `&mut self` variant for the same hook. - -The hook is **not** used for unary `prompt` / `execute`. - -**JavaScript (`@yai-xlai/xlai`):** on **`AgentSession`**, call **`registerContextCompressor(async (messages, estimatedInputTokens) => messages)`** before **`streamPrompt`** / **`streamPromptWithContent`**. The callback receives the same semantics as Rust (message array in Rust JSON shape, `estimatedInputTokens` as `number | null`). WASM exports **`registerContextCompressor`**, **`streamPrompt`**, and **`streamPromptWithContent`** on **`AgentSession`**. - -## Streaming - -The runtime supports streamed chat output through `ChatChunk` and `ChatExecutionEvent`. - -- **`Chat`**: each `stream` / `stream_prompt` / `stream_*` call performs **one** model run; events are deltas and a final `ChatChunk::Finished` for that turn. -- **`Agent`**: streaming uses the same chunk types, but the stream may include multiple model rounds while tools are still being requested. Between rounds you may see **`ChatExecutionEvent::ToolCall`** and **`ToolResult`** events after a finished assistant message that contained tool calls. An optional context-compressor hook may rewrite the message list before each of those model calls (Rust **`with_context_compressor`**, or JS **`AgentSession.registerContextCompressor`**). In JS, **`streamPrompt`** / **`streamPromptWithContent`** collect the full event list in order (one round-trip through the WASM bridge; not a browser `ReadableStream` yet). - -Streaming currently includes: - -- message start events -- content delta events -- tool call delta events -- final response events (per model turn; agent streams may emit several before the stream ends) - -## Testing Model - -The repository uses two test classes. - -### 1. Unit and mock tests - -These are the default tests and do not require API tokens. - -Run locally with: - -```bash -cargo test --workspace -``` - -### 2. End-to-end tests - -These use real provider credentials and are ignored by default. - -Run locally with: +For end-to-end provider tests, copy `.env.example` to `.env` and fill in credentials, then run: ```bash cargo test --workspace -- --ignored --test-threads=1 ``` -The current OpenAI smoke tests will also load `.env` automatically for local runs. - -For ASR/transcription end-to-end coverage, set `OPENAI_TRANSCRIPTION_MODEL` to a -transcription-capable model such as `gpt-4o-mini-transcribe`. - -## CI - -### Build workflow - -`.github/workflows/build.yml` builds: - -- Linux -- Windows -- macOS arm64 -- macOS x86_64 via cross-target build -- `wasm32-unknown-unknown` -- the `@yai-xlai/xlai` package bundle through the `pnpm` workspace - -### Test workflow - -`.github/workflows/test.yml` runs: - -- `cargo fmt --all -- --check` -- `cargo clippy --workspace --all-targets -- -D warnings` -- `cargo test --workspace` -- `pnpm --filter @yai-xlai/xlai test` - -### Publish workflow - -`.github/workflows/publish.yml` runs **crates.io** and **npm** checks: - -- On **pull requests** and pushes to **`main`**: `cargo publish --dry-run` for the [publishable Rust crate subset](docs/development/publishing.md) (in dependency order) and `npm publish --dry-run` for `@yai-xlai/xlai` after a full package build. -- On pushes to tags matching **`v*`**: real `cargo publish` and `npm publish` using secrets on the GitHub **`publish`** environment (`CRATES_IO_TOKEN`, `NPM_TOKEN`). - -See [`docs/development/publishing.md`](docs/development/publishing.md) for ordering, version bumps, and crates that are not on crates.io yet. - -### E2E workflow - -`.github/workflows/e2e.yml` runs ignored tests with provider credentials (and skips QTS model-dir tests until a CI fixture download exists). - -It is intended to use a protected GitHub Environment such as `e2e`, with maintainer approval and environment secrets. - -The OpenAI e2e environment currently expects: - -- `OPENAI_API_KEY` as a GitHub secret -- `OPENAI_BASE_URL` as a GitHub environment variable -- `OPENAI_MODEL` as a GitHub environment variable -- `OPENAI_TRANSCRIPTION_MODEL` as a GitHub environment variable - -The ignored native `llama.cpp` smoke test expects: - -- `LLAMA_CPP_MODEL` pointing to a local GGUF model file, or the default fixture path `fixtures/llama.cpp/Qwen3.5-0.8B-Q4_0.gguf` - -The repository intentionally ignores downloaded GGUF fixtures under `fixtures/llama.cpp/`. -CI downloads that fixture with the Hugging Face CLI and caches it between runs. - -Ignored QTS integration tests in `xlai-qts-core` expect a full Qwen3 TTS model directory: - -- `XLAI_QTS_MODEL_DIR` pointing at a folder containing the GGUF talker, ONNX vocoder (`qwen3-tts-vocoder.onnx`), tokenizer, and `config.json` (see `xlai_qts_core::ModelPaths`). For **ICL** voice clone, also add `qwen3-tts-reference-codec.onnx` and `qwen3-tts-reference-codec-psreprocess.json` (export with `uv run export-model-artifacts`; see `docs/qts/export-and-hf-publish.md`). - -CI e2e currently **skips** those tests because no Hugging Face download step is wired yet; run them locally after downloading weights. - -To build `xlai-native` with the QTS backend available, enable `--features qts` (optional; avoids linking QTS/ggml in default builds). - -## Current Design Notes - -- unified API across native and browser targets -- target-specific dependency wiring for native vs `wasm32` -- OpenAI-compatible backend uses `reqwest` -- public APIs aim to be async-first and streaming-capable - -## Roadmap - -Planned or expected next areas include: - -- more provider backends -- typed tool argument helpers -- embeddings and retrieval backends -- skill management APIs -- local/device inference backends -- more browser-focused examples +See [`docs/guide/configuration.md`](docs/guide/configuration.md) for the full list of environment variables and [`docs/development/ci-and-testing.md`](docs/development/ci-and-testing.md) for the testing model. ## Support of LLM API Providers ### Chat API -- [x] OpenAI Backends - - [x] [AIHubMix](https://aihubmix.com/?aff=OOiX) - - [x] [OpenAI](https://platform.openai.com/docs/guides/gpt/chat-completions-api) +- [x] OpenAI + - [x] [OpenAI](https://developers.openai.com/api/docs/guides/text) - [x] [Azure OpenAI API](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference) -- [x] OpenRouter Backend -- [x] llama.cpp Backends +- [x] [OpenRouter](https://openrouter.ai/docs/api/api-reference/responses/create-responses) +- [x] llama.cpp - [x] CPU - [x] OpenBLAS - [x] Accelerate.framework @@ -525,4 +94,4 @@ Planned or expected next areas include: ## License -Apache License 2.0. See `LICENSE`. +Apache License 2.0. See [`LICENSE`](LICENSE). From 96178249ada54d036bd58a6edc2467c76f060617 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 21:50:13 +0800 Subject: [PATCH 02/12] update docs --- ARCHITECTURE.md | 4 +- README.md | 38 ++++++++------- .../xlai-backend-llama-cpp/Cargo.toml | 5 +- crates/platform/xlai-facade/Cargo.toml | 3 ++ crates/platform/xlai-ffi/Cargo.toml | 5 +- crates/platform/xlai-native/Cargo.toml | 5 +- crates/qts/xlai-qts-cli/Cargo.toml | 3 +- crates/qts/xlai-qts-core/Cargo.toml | 9 ++-- .../sys/xlai-build-native/src/ggml_build.rs | 10 +++- crates/sys/xlai-sys-ggml/build.rs | 15 +++--- crates/sys/xlai-sys-llama/Cargo.toml | 3 ++ crates/sys/xlai-sys-llama/build.rs | 48 +++++++++++++++++-- docs/architecture/index.md | 4 +- docs/development/native-vendor.md | 15 ++++++ docs/guide/getting-started.md | 2 + docs/providers/index.md | 2 +- docs/rust/index.md | 12 ++--- 17 files changed, 136 insertions(+), 47 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 09a10ae..d1c3069 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -21,8 +21,8 @@ Workspace crates are grouped under `crates/core/`, `crates/runtime/`, `crates/ba | `xlai-sys-llama` | Vendored `llama.cpp` build (CMake + bindgen) for the llama backend. Sources: `vendor/native/llama.cpp`. | | `xlai-sys-ggml` | Vendored standalone `ggml` build (CMake + bindgen) for QTS. Sources: `vendor/native/ggml`. | | `xlai-qts-cli` | `synthesize` / `profile` / `tui` binary (`xlai-qts`) for local TTS workflows. | -| `xlai-facade` | Internal (not on crates.io): native-only integration layer used by **`xlai-native`** (re-exports, optional `llama` / `qts`, Gemini + OpenAI + transformers.js for the aggregate). **`xlai-wasm` does not depend on it**; WASM re-exports come from `xlai-core`, `xlai-runtime`, and backends directly. | -| `xlai-native` | Native Rust entrypoint: **explicit** public re-exports (plus `gemini` submodule for workspace-only Gemini types). Uses `xlai-facade` for feature wiring. Enable optional `qts` for `QtsTtsModel` (avoids linking QTS/ggml unless needed). | +| `xlai-facade` | Internal (not on crates.io): native-only integration layer used by **`xlai-native`** (re-exports, default local `llama.cpp` wiring plus optional `qts`, Gemini + OpenAI + transformers.js for the aggregate). **`xlai-wasm` does not depend on it**; WASM re-exports come from `xlai-core`, `xlai-runtime`, and backends directly. | +| `xlai-native` | Native Rust entrypoint: **explicit** public re-exports (plus `gemini` submodule for workspace-only Gemini types). Uses `xlai-facade` for feature wiring. Local `llama.cpp` is included by default; enable optional `qts` for `QtsTtsModel` (avoids linking QTS/ggml unless needed). | | `xlai-wasm` | `wasm-bindgen` entry points and JS-facing session factories. Public Rust types are re-exported from `xlai-core` / `xlai-runtime` / OpenAI + (on wasm32) transformers.js — no `xlai-facade` dependency. Default feature `qts` enables local QTS WASM surface (stub `TtsModel`, shared browser manifest types, `qtsBrowserTts*`; see `docs/qts/wasm-browser-runtime.md`). | | `xlai-ffi` | C ABI facade for future native interop. | diff --git a/README.md b/README.md index 312fefa..6e123a6 100644 --- a/README.md +++ b/README.md @@ -8,18 +8,18 @@ The full documentation lives in the VitePress site under [`docs/`](docs/): -| Topic | Link | -| ---------------------- | ----------------------------------------------------------- | -| Introduction | [`docs/guide/index.md`](docs/guide/index.md) | -| Install and build | [`docs/guide/getting-started.md`](docs/guide/getting-started.md) | -| Environment variables | [`docs/guide/configuration.md`](docs/guide/configuration.md) | -| Architecture & crates | [`docs/architecture/index.md`](docs/architecture/index.md) | -| Crate taxonomy | [`docs/development/crates-taxonomy.md`](docs/development/crates-taxonomy.md) | -| Rust SDK | [`docs/rust/index.md`](docs/rust/index.md) | -| JavaScript / WASM | [`docs/js/index.md`](docs/js/index.md) | -| Provider support | [`docs/providers/index.md`](docs/providers/index.md) | -| QTS (Qwen3 TTS) | [`docs/qts/index.md`](docs/qts/index.md) | -| CI, testing, releases | [`docs/development/index.md`](docs/development/index.md) | +| Topic | Link | +| --------------------- | ---------------------------------------------------------------------------- | +| Introduction | [`docs/guide/index.md`](docs/guide/index.md) | +| Install and build | [`docs/guide/getting-started.md`](docs/guide/getting-started.md) | +| Environment variables | [`docs/guide/configuration.md`](docs/guide/configuration.md) | +| Architecture & crates | [`docs/architecture/index.md`](docs/architecture/index.md) | +| Crate taxonomy | [`docs/development/crates-taxonomy.md`](docs/development/crates-taxonomy.md) | +| Rust SDK | [`docs/rust/index.md`](docs/rust/index.md) | +| JavaScript / WASM | [`docs/js/index.md`](docs/js/index.md) | +| Provider support | [`docs/providers/index.md`](docs/providers/index.md) | +| QTS (Qwen3 TTS) | [`docs/qts/index.md`](docs/qts/index.md) | +| CI, testing, releases | [`docs/development/index.md`](docs/development/index.md) | Preview the site locally: @@ -39,6 +39,8 @@ cargo build --workspace cargo test --workspace ``` +Native builds now default to `openblas`, `cuda`, `hip`, and `openvino` for the local `llama.cpp` / QTS stacks. On unsupported Apple targets those accelerator flags are skipped with build warnings; on supported Linux/Windows hosts, install the corresponding SDK/toolchain before building. + JavaScript workspace: ```bash @@ -69,9 +71,9 @@ See [`docs/guide/configuration.md`](docs/guide/configuration.md) for the full li - [x] Accelerate.framework - [x] Metal - [x] Vulkan - - [ ] CUDA - - [ ] HIP - - [ ] OpenVINO + - [x] CUDA + - [x] HIP + - [x] OpenVINO ### ASR API @@ -87,9 +89,9 @@ See [`docs/guide/configuration.md`](docs/guide/configuration.md) for the full li - [x] Accelerate.framework - [x] Metal - [x] Vulkan - - [ ] CUDA - - [ ] HIP - - [ ] OpenVINO + - [x] CUDA + - [x] HIP + - [x] OpenVINO - Vocoder (ORT default EPs) ## License diff --git a/crates/backends/xlai-backend-llama-cpp/Cargo.toml b/crates/backends/xlai-backend-llama-cpp/Cargo.toml index 8eeb1ce..a649e93 100644 --- a/crates/backends/xlai-backend-llama-cpp/Cargo.toml +++ b/crates/backends/xlai-backend-llama-cpp/Cargo.toml @@ -9,10 +9,13 @@ description = "llama.cpp-backed chat and embeddings backend for xlai (depends on publish = false [features] -default = ["openblas"] +default = ["openblas", "cuda", "hip", "openvino"] openblas = ["xlai-sys-llama/openblas"] metal = ["xlai-sys-llama/metal"] vulkan = ["xlai-sys-llama/vulkan"] +cuda = ["xlai-sys-llama/cuda"] +hip = ["xlai-sys-llama/hip"] +openvino = ["xlai-sys-llama/openvino"] [dependencies] async-stream.workspace = true diff --git a/crates/platform/xlai-facade/Cargo.toml b/crates/platform/xlai-facade/Cargo.toml index eef7b08..5e703a0 100644 --- a/crates/platform/xlai-facade/Cargo.toml +++ b/crates/platform/xlai-facade/Cargo.toml @@ -19,6 +19,9 @@ llama = ["dep:xlai-backend-llama-cpp"] openblas = ["llama", "xlai-backend-llama-cpp/openblas", "xlai-qts-core?/openblas"] metal = ["llama", "xlai-backend-llama-cpp/metal", "xlai-qts-core?/metal"] vulkan = ["llama", "xlai-backend-llama-cpp/vulkan", "xlai-qts-core?/vulkan"] +cuda = ["llama", "xlai-backend-llama-cpp/cuda", "xlai-qts-core?/cuda"] +hip = ["llama", "xlai-backend-llama-cpp/hip", "xlai-qts-core?/hip"] +openvino = ["llama", "xlai-backend-llama-cpp/openvino", "xlai-qts-core?/openvino"] qts = ["dep:xlai-qts-core"] [dependencies] diff --git a/crates/platform/xlai-ffi/Cargo.toml b/crates/platform/xlai-ffi/Cargo.toml index c4551a8..6b69963 100644 --- a/crates/platform/xlai-ffi/Cargo.toml +++ b/crates/platform/xlai-ffi/Cargo.toml @@ -9,10 +9,13 @@ description = "C ABI shared library wrapping `xlai-native`." publish = false [features] -default = ["openblas"] +default = ["openblas", "cuda", "hip", "openvino"] openblas = ["xlai-native/openblas"] metal = ["xlai-native/metal"] vulkan = ["xlai-native/vulkan"] +cuda = ["xlai-native/cuda"] +hip = ["xlai-native/hip"] +openvino = ["xlai-native/openvino"] [lib] crate-type = ["cdylib"] diff --git a/crates/platform/xlai-native/Cargo.toml b/crates/platform/xlai-native/Cargo.toml index 21d2087..5f45849 100644 --- a/crates/platform/xlai-native/Cargo.toml +++ b/crates/platform/xlai-native/Cargo.toml @@ -10,10 +10,13 @@ publish = false build = "build.rs" [features] -default = ["openblas"] +default = ["openblas", "cuda", "hip", "openvino"] openblas = ["xlai-facade/openblas"] metal = ["xlai-facade/metal"] vulkan = ["xlai-facade/vulkan"] +cuda = ["xlai-facade/cuda"] +hip = ["xlai-facade/hip"] +openvino = ["xlai-facade/openvino"] # Optional: avoid linking QTS (standalone GGML) unless you need native TTS in the aggregate crate. qts = ["xlai-facade/qts"] diff --git a/crates/qts/xlai-qts-cli/Cargo.toml b/crates/qts/xlai-qts-cli/Cargo.toml index 71a6dc3..c18e0cf 100644 --- a/crates/qts/xlai-qts-cli/Cargo.toml +++ b/crates/qts/xlai-qts-cli/Cargo.toml @@ -14,7 +14,7 @@ name = "xlai-qts" path = "src/main.rs" [features] -default = ["openblas"] +default = ["openblas", "cuda", "hip", "openvino"] openblas = ["xlai-qts-core/openblas"] metal = ["xlai-qts-core/metal"] vulkan = ["xlai-qts-core/vulkan"] @@ -31,6 +31,7 @@ onednn = ["xlai-qts-core/onednn"] openvino = ["xlai-qts-core/openvino"] qnn = ["xlai-qts-core/qnn"] rknpu = ["xlai-qts-core/rknpu"] +hip = ["xlai-qts-core/hip"] rocm = ["xlai-qts-core/rocm"] tensorrt = ["xlai-qts-core/tensorrt"] tvm = ["xlai-qts-core/tvm"] diff --git a/crates/qts/xlai-qts-core/Cargo.toml b/crates/qts/xlai-qts-core/Cargo.toml index 0494a08..19d541e 100644 --- a/crates/qts/xlai-qts-core/Cargo.toml +++ b/crates/qts/xlai-qts-core/Cargo.toml @@ -9,7 +9,7 @@ description = "Qwen3 TTS inference (GGUF + GGML) — engine and native TtsModel publish = false [features] -default = ["openblas"] +default = ["openblas", "cuda", "hip", "openvino"] metal = ["xlai-sys-ggml/metal"] vulkan = ["xlai-sys-ggml/vulkan"] native = ["xlai-sys-ggml/native"] @@ -18,15 +18,16 @@ acl = ["ort/acl"] armnn = ["ort/armnn"] cann = ["ort/cann"] coreml = ["ort/coreml"] -cuda = ["ort/cuda"] +cuda = ["xlai-sys-ggml/cuda", "ort/cuda"] directml = ["ort/directml"] migraphx = ["ort/migraphx"] nnapi = ["ort/nnapi"] onednn = ["ort/onednn"] -openvino = ["ort/openvino"] +openvino = ["xlai-sys-ggml/openvino", "ort/openvino"] qnn = ["ort/qnn"] rknpu = ["ort/rknpu"] -rocm = ["ort/rocm"] +hip = ["xlai-sys-ggml/hip"] +rocm = ["hip", "ort/rocm"] tensorrt = ["ort/tensorrt"] tvm = ["ort/tvm"] vitis = ["ort/vitis"] diff --git a/crates/sys/xlai-build-native/src/ggml_build.rs b/crates/sys/xlai-build-native/src/ggml_build.rs index ee160ef..a24d73e 100644 --- a/crates/sys/xlai-build-native/src/ggml_build.rs +++ b/crates/sys/xlai-build-native/src/ggml_build.rs @@ -63,7 +63,15 @@ pub fn validate_ggml_features(target: &str, crate_label: &str) { ); } if feature_enabled("cuda") && target.contains("apple") { - panic!("{crate_label}: `cuda` feature is not supported on Apple targets"); + println!("cargo:warning={crate_label}: `cuda` feature ignored on Apple target ({target})"); + } + if feature_enabled("hip") && target.contains("apple") { + println!("cargo:warning={crate_label}: `hip` feature ignored on Apple target ({target})"); + } + if feature_enabled("openvino") && target.contains("apple") { + println!( + "cargo:warning={crate_label}: `openvino` feature ignored on Apple target ({target})" + ); } } diff --git a/crates/sys/xlai-sys-ggml/build.rs b/crates/sys/xlai-sys-ggml/build.rs index 582aa96..e381346 100644 --- a/crates/sys/xlai-sys-ggml/build.rs +++ b/crates/sys/xlai-sys-ggml/build.rs @@ -22,6 +22,9 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { let target = env::var("TARGET").expect("TARGET"); let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default(); let openblas_fe = feature_enabled("openblas"); + let enable_cuda = feature_enabled("cuda") && !target.contains("apple"); + let enable_hip = feature_enabled("hip") && !target.contains("apple"); + let enable_openvino = feature_enabled("openvino") && !target.contains("apple"); let enable_linux_windows_blas = openblas_fe && matches!(target_os.as_str(), "linux" | "windows"); let link_ggml_blas = openblas_fe && (enable_linux_windows_blas || target.contains("apple")); @@ -111,15 +114,15 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { apply_cmake_env_overrides(&mut cfg, enable_linux_windows_blas); - map_feature_cmake(&mut cfg, "cuda", "GGML_CUDA"); + cfg.define("GGML_CUDA", if enable_cuda { "ON" } else { "OFF" }); map_feature_cmake(&mut cfg, "vulkan", "GGML_VULKAN"); - map_feature_cmake(&mut cfg, "hip", "GGML_HIP"); + cfg.define("GGML_HIP", if enable_hip { "ON" } else { "OFF" }); map_feature_cmake(&mut cfg, "musa", "GGML_MUSA"); map_feature_cmake(&mut cfg, "opencl", "GGML_OPENCL"); map_feature_cmake(&mut cfg, "rpc", "GGML_RPC"); map_feature_cmake(&mut cfg, "sycl", "GGML_SYCL"); map_feature_cmake(&mut cfg, "webgpu", "GGML_WEBGPU"); - map_feature_cmake(&mut cfg, "openvino", "GGML_OPENVINO"); + cfg.define("GGML_OPENVINO", if enable_openvino { "ON" } else { "OFF" }); map_feature_cmake(&mut cfg, "hexagon", "GGML_HEXAGON"); map_feature_cmake(&mut cfg, "cann", "GGML_CANN"); map_feature_cmake(&mut cfg, "zendnn", "GGML_ZENDNN"); @@ -139,14 +142,14 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { if feature_enabled("metal") && target.contains("apple") { println!("cargo:rustc-link-lib=static=ggml-metal"); } - if feature_enabled("cuda") { + if enable_cuda { println!("cargo:rustc-link-lib=static=ggml-cuda"); } if feature_enabled("vulkan") { println!("cargo:rustc-link-lib=static=ggml-vulkan"); emit_vulkan_loader_links(&target); } - if feature_enabled("hip") { + if enable_hip { println!("cargo:rustc-link-lib=static=ggml-hip"); } if feature_enabled("musa") { @@ -167,7 +170,7 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { if feature_enabled("webgpu") { println!("cargo:rustc-link-lib=static=ggml-webgpu"); } - if feature_enabled("openvino") { + if enable_openvino { println!("cargo:rustc-link-lib=static=ggml-openvino"); } if feature_enabled("hexagon") { diff --git a/crates/sys/xlai-sys-llama/Cargo.toml b/crates/sys/xlai-sys-llama/Cargo.toml index 8d87b55..f7d87c9 100644 --- a/crates/sys/xlai-sys-llama/Cargo.toml +++ b/crates/sys/xlai-sys-llama/Cargo.toml @@ -21,6 +21,9 @@ default = [] openblas = [] metal = [] vulkan = [] +cuda = [] +hip = [] +openvino = [] [build-dependencies] bindgen.workspace = true diff --git a/crates/sys/xlai-sys-llama/build.rs b/crates/sys/xlai-sys-llama/build.rs index f81d1bd..e35c947 100644 --- a/crates/sys/xlai-sys-llama/build.rs +++ b/crates/sys/xlai-sys-llama/build.rs @@ -25,14 +25,23 @@ struct BackendFeatureSet { openblas: bool, metal: bool, vulkan: bool, + cuda: bool, + hip: bool, + openvino: bool, } impl BackendFeatureSet { fn from_cargo_features(target_os: &str) -> BuildResult { + let requested_cuda = feature_enabled("cuda"); + let requested_hip = feature_enabled("hip"); + let requested_openvino = feature_enabled("openvino"); let feature_set = Self { openblas: feature_enabled("openblas"), metal: feature_enabled("metal"), vulkan: feature_enabled("vulkan"), + cuda: requested_cuda && target_os != "macos", + hip: requested_hip && target_os != "macos", + openvino: requested_openvino && target_os != "macos", }; if feature_set.metal && target_os != "macos" { @@ -41,6 +50,15 @@ impl BackendFeatureSet { ) .into()); } + if requested_cuda && target_os == "macos" { + println!("cargo:warning=xlai-sys-llama: `cuda` feature ignored on macOS targets"); + } + if requested_hip && target_os == "macos" { + println!("cargo:warning=xlai-sys-llama: `hip` feature ignored on macOS targets"); + } + if requested_openvino && target_os == "macos" { + println!("cargo:warning=xlai-sys-llama: `openvino` feature ignored on macOS targets"); + } Ok(feature_set) } @@ -50,6 +68,9 @@ impl BackendFeatureSet { self.openblas.then_some("openblas"), self.metal.then_some("metal"), self.vulkan.then_some("vulkan"), + self.cuda.then_some("cuda"), + self.hip.then_some("hip"), + self.openvino.then_some("openvino"), ] .into_iter() .flatten() @@ -134,11 +155,14 @@ fn build_llama_cpp_stack(manifest_dir: &Path) -> BuildResult<()> { .define("GGML_METAL", if feature_set.metal { "ON" } else { "OFF" }) .define("GGML_RPC", "OFF") .define("GGML_VULKAN", if feature_set.vulkan { "ON" } else { "OFF" }) - .define("GGML_CUDA", "OFF") - .define("GGML_HIP", "OFF") + .define("GGML_CUDA", if feature_set.cuda { "ON" } else { "OFF" }) + .define("GGML_HIP", if feature_set.hip { "ON" } else { "OFF" }) .define("GGML_SYCL", "OFF") .define("GGML_OPENCL", "OFF") - .define("GGML_OPENVINO", "OFF") + .define( + "GGML_OPENVINO", + if feature_set.openvino { "ON" } else { "OFF" }, + ) .define("GGML_ZDNN", "OFF") .define("GGML_VIRTGPU", "OFF") .define("GGML_WEBGPU", "OFF"); @@ -193,6 +217,15 @@ fn build_llama_cpp_stack(manifest_dir: &Path) -> BuildResult<()> { if feature_set.vulkan { libraries.push("ggml-vulkan"); } + if feature_set.cuda { + libraries.push("ggml-cuda"); + } + if feature_set.hip { + libraries.push("ggml-hip"); + } + if feature_set.openvino { + libraries.push("ggml-openvino"); + } for library in libraries { println!("cargo:rustc-link-lib=static={library}"); } @@ -276,6 +309,15 @@ fn emit_llama_search_paths(dst: &Path, feature_set: BackendFeatureSet, enable_op if feature_set.vulkan { emit_search_path_variants(&ggml_src_dir.join("ggml-vulkan")); } + if feature_set.cuda { + emit_search_path_variants(&ggml_src_dir.join("ggml-cuda")); + } + if feature_set.hip { + emit_search_path_variants(&ggml_src_dir.join("ggml-hip")); + } + if feature_set.openvino { + emit_search_path_variants(&ggml_src_dir.join("ggml-openvino")); + } } fn llguidance_output_dir(dst: &Path) -> PathBuf { diff --git a/docs/architecture/index.md b/docs/architecture/index.md index 939f23c..a835b4c 100644 --- a/docs/architecture/index.md +++ b/docs/architecture/index.md @@ -24,8 +24,8 @@ The same content lives in the repository as [`ARCHITECTURE.md`](https://github.c | `xlai-sys-llama` | Vendored `llama.cpp` build (CMake + bindgen) for the llama backend (`vendor/native/llama.cpp`). | | `xlai-sys-ggml` | Vendored standalone `ggml` build (CMake + bindgen) for QTS (`vendor/native/ggml`). | | `xlai-qts-cli` | `synthesize` / `profile` / `tui` binary (`xlai-qts`) for local TTS workflows. | -| `xlai-facade` | Internal (not on crates.io): **native-only** integration layer for `xlai-native` (optional `llama` / `qts`, Gemini + HTTP/local backends). `xlai-wasm` does not use this crate. | -| `xlai-native` | Native Rust entrypoint: explicit re-exports and `gemini` submodule; uses `xlai-facade` for feature wiring. Enable optional `qts` for `QtsTtsModel`. | +| `xlai-facade` | Internal (not on crates.io): **native-only** integration layer for `xlai-native` (default local `llama.cpp` wiring plus optional `qts`, Gemini + HTTP/local backends). `xlai-wasm` does not use this crate. | +| `xlai-native` | Native Rust entrypoint: explicit re-exports and `gemini` submodule; uses `xlai-facade` for feature wiring. Local `llama.cpp` is included by default; enable optional `qts` for `QtsTtsModel`. | | `xlai-wasm` | `wasm-bindgen` entry points and JS-facing session factories. Re-exports from `xlai-core` / `xlai-runtime` / backends (no `xlai-facade`). Default feature `qts` enables local QTS WASM surface (stub `TtsModel`, shared browser manifest types, `qtsBrowserTts*`; see [Browser / WASM runtime](/qts/wasm-browser-runtime)). | | `xlai-ffi` | C ABI facade for future native interop. | diff --git a/docs/development/native-vendor.md b/docs/development/native-vendor.md index 0a79401..47371e4 100644 --- a/docs/development/native-vendor.md +++ b/docs/development/native-vendor.md @@ -16,6 +16,21 @@ git submodule update --init --recursive vendor/native/llama.cpp vendor/native/gg - **`GGML_SRC`**: absolute path to a `ggml` checkout for `xlai-sys-ggml` (defaults to `vendor/native/ggml`). - **`LLAMA_CPP_SRC`**: absolute path to a `llama.cpp` checkout for `xlai-sys-llama` (defaults to `vendor/native/llama.cpp`). +## Default accelerator set + +The native `llama.cpp` / QTS crates now default to: + +- `openblas` +- `cuda` +- `hip` +- `openvino` + +Behavior is platform-dependent: + +- On unsupported Apple targets, `cuda`, `hip`, and `openvino` are skipped with build warnings so default local builds still work. +- On supported Linux/Windows targets, those defaults mean the corresponding SDKs/toolchains must be available when you build the native crates. +- Vulkan and Metal remain opt-in feature flags. + ## Dual native stacks Enabling both local chat (`xlai-sys-llama`, which bundles `ggml` with `llama.cpp`) and native QTS (`xlai-sys-ggml`) links **two** native `ggml` implementations into one binary. Build scripts emit a `cargo:warning` when `xlai-facade` has `llama` + `qts`, or when `xlai-native` enables `qts`. Prefer separate processes or a single stack if you hit duplicate symbols or linker issues. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index eb6583a..86b049a 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -16,6 +16,8 @@ cd xlai cargo build --workspace ``` +Native crates that wrap `llama.cpp` / QTS now default to `openblas`, `cuda`, `hip`, and `openvino`. On unsupported Apple targets those accelerator flags are ignored with warnings; on supported Linux/Windows hosts you should have the relevant CUDA, ROCm/HIP, and OpenVINO toolchains installed before building. + ## Tests ```bash diff --git a/docs/providers/index.md b/docs/providers/index.md index 840d5d5..0a088af 100644 --- a/docs/providers/index.md +++ b/docs/providers/index.md @@ -34,4 +34,4 @@ OpenRouter remains chat-only in XLAI today. The dedicated `xlai-backend-openrout ## llama.cpp acceleration matrix -The README tracks CPU/GPU EP support (Metal, Vulkan, CUDA roadmap, etc.). See [Support of LLM API providers](https://github.com/yetanother.ai/xlai/blob/main/README.md#support-of-llm-api-providers) on GitHub for the live checklist. +The README tracks the live native accelerator matrix for `llama.cpp` and QTS. Today the workspace defaults the local native stacks to `openblas`, `cuda`, `hip`, and `openvino`, while Vulkan and Metal remain opt-in. See [Support of LLM API providers](https://github.com/yetanother.ai/xlai/blob/main/README.md#support-of-llm-api-providers) on GitHub for the checklist. diff --git a/docs/rust/index.md b/docs/rust/index.md index abd91f4..aaebdab 100644 --- a/docs/rust/index.md +++ b/docs/rust/index.md @@ -4,11 +4,11 @@ Application code typically depends on **`xlai-native`** (or **`xlai-wasm`** in t ## Platform entrypoints -| Crate | Use when | -| ------------- | --------------------------------------------------------------------------------------------- | -| `xlai-native` | Native binaries and servers on macOS, Linux, or Windows. Optional `llama` and `qts` features. | -| `xlai-wasm` | `wasm32-unknown-unknown` builds and JavaScript interop. | -| `xlai-ffi` | C ABI / shared library embedding (wraps `xlai-native`). | +| Crate | Use when | +| ------------- | --------------------------------------------------------------------------------------------------------------------- | +| `xlai-native` | Native binaries and servers on macOS, Linux, or Windows. Includes local `llama.cpp` by default; `qts` stays optional. | +| `xlai-wasm` | `wasm32-unknown-unknown` builds and JavaScript interop. | +| `xlai-ffi` | C ABI / shared library embedding (wraps `xlai-native`). | Domain types and traits live in **`xlai-core`** (semver-stable on crates.io). Session APIs live in **`xlai-runtime`** and are re-exported through **`xlai-native`** / **`xlai-wasm`**. The **`xlai-facade`** crate is an internal workspace helper (not on crates.io) used **only by `xlai-native`** for native aggregate wiring; **`xlai-wasm` does not depend on it**. @@ -27,4 +27,4 @@ Concrete examples (OpenAI, llama.cpp, streaming agents) are in the repository [R ## Optional: QTS in native builds -Enable the **`qts`** feature on `xlai-native` when you need `QtsTtsModel` without taking the dependency in every build. +`xlai-native` defaults the local `llama.cpp` stack to `openblas`, `cuda`, `hip`, and `openvino`. Enable the **`qts`** feature when you need `QtsTtsModel`; QTS follows the same default accelerator set, while unsupported Apple-only combinations are skipped with warnings at build time. From 6e1b3c87a2d4154d91af4ffc43e3330b442596b6 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 21:59:07 +0800 Subject: [PATCH 03/12] update cuda toolkit support --- .github/actions/setup-xlai-rust-native/action.yml | 12 ++++++++++-- docs/development/ci-and-testing.md | 4 ++++ docs/development/crates-taxonomy.md | 2 +- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index ed81989..8fe6108 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -1,8 +1,8 @@ -# Shared steps for native Rust jobs: toolchain, sccache, rust-cache, OpenBLAS / vcpkg / shaderc. +# Shared steps for native Rust jobs: toolchain, sccache, rust-cache, CUDA, OpenBLAS / vcpkg / shaderc. # Call after `actions/checkout` (with `submodules: true` when building xlai-sys-llama or xlai-sys-ggml; submodules live under `vendor/native/`). name: XLAI Rust native setup -description: Rust toolchain, sccache, cache, and OS-specific native dependencies +description: Rust toolchain, sccache, cache, CUDA, and OS-specific native dependencies inputs: rust-targets: @@ -52,6 +52,14 @@ runs: path: ${{ env.VCPKG_ROOT }} key: vcpkg-openblas-${{ runner.os }}-${{ env.VCPKG_OPENBLAS_TRIPLET }}-v1 + - name: Install CUDA toolkit + if: runner.os == 'Linux' || runner.os == 'Windows' + uses: Jimver/cuda-toolkit@v0.2 + with: + cuda: '13.2.0' + method: network + log-file-suffix: ${{ format('{0}-{1}.txt', runner.os, inputs.rust-cache-shared-key) }} + - name: Install Linux build dependencies if: runner.os == 'Linux' shell: bash diff --git a/docs/development/ci-and-testing.md b/docs/development/ci-and-testing.md index fb34b34..3eb0ac8 100644 --- a/docs/development/ci-and-testing.md +++ b/docs/development/ci-and-testing.md @@ -26,6 +26,8 @@ OpenAI smoke tests load `.env` automatically for local runs. For embeddings/tran Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unknown-unknown`, and the `@yai-xlai/xlai` package through the pnpm workspace. +Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, and now installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. + ### Test (`.github/workflows/test.yml`) - `cargo fmt --all -- --check` @@ -33,6 +35,8 @@ Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unkn - `cargo test --workspace` - `pnpm --filter @yai-xlai/xlai test` +The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit before building the default native feature set. + ### Publish (`.github/workflows/publish.yml`) On pull requests and pushes to `main`: `cargo publish --dry-run` for the publishable Rust crate subset and `npm publish --dry-run` for `@yai-xlai/xlai`. diff --git a/docs/development/crates-taxonomy.md b/docs/development/crates-taxonomy.md index d7c51ab..c86f7bf 100644 --- a/docs/development/crates-taxonomy.md +++ b/docs/development/crates-taxonomy.md @@ -42,7 +42,7 @@ This doc classifies crates for dependency and release decisions. Authoritative p ## CI setup -Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, OpenBLAS / vcpkg / shaderc). Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. +Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, plus OpenBLAS / vcpkg / shaderc). Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. ## Adding a new workspace member From 8c9b07f580ca85d6f84705531581b06d49b197f9 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 22:00:15 +0800 Subject: [PATCH 04/12] fixes --- .github/actions/setup-xlai-rust-native/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index 8fe6108..a53e840 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -54,7 +54,7 @@ runs: - name: Install CUDA toolkit if: runner.os == 'Linux' || runner.os == 'Windows' - uses: Jimver/cuda-toolkit@v0.2 + uses: Jimver/cuda-toolkit@v0.2.57 with: cuda: '13.2.0' method: network From 3d9e453287e535433564a2c069ce5f09f73c4f19 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 22:01:06 +0800 Subject: [PATCH 05/12] ag --- .github/actions/setup-xlai-rust-native/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index a53e840..07e231a 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -54,7 +54,7 @@ runs: - name: Install CUDA toolkit if: runner.os == 'Linux' || runner.os == 'Windows' - uses: Jimver/cuda-toolkit@v0.2.57 + uses: Jimver/cuda-toolkit@v0.2.35 with: cuda: '13.2.0' method: network From 6270748e533f494fb0faaa6721602c2012de7aca Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 22:21:28 +0800 Subject: [PATCH 06/12] add rocm download --- .../actions/setup-xlai-rust-native/action.yml | 94 ++++++++++++++++++- docs/development/ci-and-testing.md | 4 +- docs/development/crates-taxonomy.md | 2 +- 3 files changed, 95 insertions(+), 5 deletions(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index 07e231a..505577a 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -1,8 +1,8 @@ -# Shared steps for native Rust jobs: toolchain, sccache, rust-cache, CUDA, OpenBLAS / vcpkg / shaderc. +# Shared steps for native Rust jobs: toolchain, sccache, rust-cache, CUDA, ROCm/HIP, OpenBLAS / vcpkg / shaderc. # Call after `actions/checkout` (with `submodules: true` when building xlai-sys-llama or xlai-sys-ggml; submodules live under `vendor/native/`). name: XLAI Rust native setup -description: Rust toolchain, sccache, cache, CUDA, and OS-specific native dependencies +description: Rust toolchain, sccache, cache, CUDA, ROCm/HIP, and OS-specific native dependencies inputs: rust-targets: @@ -60,6 +60,96 @@ runs: method: network log-file-suffix: ${{ format('{0}-{1}.txt', runner.os, inputs.rust-cache-shared-key) }} + - name: Install Linux ROCm / HIP SDK + if: runner.os == 'Linux' + shell: bash + run: | + set -euxo pipefail + + source /etc/os-release + case "${VERSION_CODENAME}" in + jammy|noble) ;; + *) + echo "Unsupported Ubuntu codename for ROCm apt install: ${VERSION_CODENAME}" >&2 + exit 1 + ;; + esac + + ROCM_APT_VERSION=7.2.2 + GRAPHICS_APT_VERSION=7.2.1 + + sudo mkdir --parents --mode=0755 /etc/apt/keyrings + wget https://repo.radeon.com/rocm/rocm.gpg.key -O - \ + | gpg --dearmor \ + | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null + + cat <> "$GITHUB_ENV" + echo "HIP_PATH=/opt/rocm" >> "$GITHUB_ENV" + echo "HIPCXX=/opt/rocm/bin/hipcc" >> "$GITHUB_ENV" + if [ -n "${CMAKE_PREFIX_PATH:-}" ]; then + echo "CMAKE_PREFIX_PATH=/opt/rocm:${CMAKE_PREFIX_PATH}" >> "$GITHUB_ENV" + else + echo "CMAKE_PREFIX_PATH=/opt/rocm" >> "$GITHUB_ENV" + fi + echo "/opt/rocm/bin" >> "$GITHUB_PATH" + + - name: Install Windows HIP SDK + if: runner.os == 'Windows' + shell: pwsh + run: | + $installerName = 'AMD-Software-PRO-Edition-25.Q3-WinSvr2022-For-HIP.exe' + $installerUrl = "https://drivers.amd.com/drivers/prographics/$installerName" + $installerPath = Join-Path $env:RUNNER_TEMP $installerName + $installLog = Join-Path $env:RUNNER_TEMP 'hip-sdk-install.log' + + Write-Host "Attempting HIP SDK install from $installerUrl" + Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath -Headers @{ + 'User-Agent' = 'Mozilla/5.0' + 'Referer' = 'https://www.amd.com/' + } + + $process = Start-Process -FilePath $installerPath ` + -ArgumentList '-install', '-log', $installLog ` + -NoNewWindow ` + -Wait ` + -PassThru + + if ($process.ExitCode -notin 0, 3010) { + throw "HIP SDK installer failed with exit code $($process.ExitCode). See $installLog" + } + + $rocmRoot = Get-ChildItem 'C:\Program Files\AMD\ROCm' -Directory ` + | Sort-Object Name -Descending ` + | Select-Object -First 1 + + if (-not $rocmRoot) { + throw 'HIP SDK install completed but no ROCm root was found under C:\Program Files\AMD\ROCm' + } + + Add-Content -Path $env:GITHUB_ENV -Value "ROCM_PATH=$($rocmRoot.FullName)" + Add-Content -Path $env:GITHUB_ENV -Value "HIP_PATH=$($rocmRoot.FullName)" + Add-Content -Path $env:GITHUB_ENV -Value "HIPCXX=$($rocmRoot.FullName)\\bin\\hipcc.exe" + if ($env:CMAKE_PREFIX_PATH) { + Add-Content -Path $env:GITHUB_ENV -Value "CMAKE_PREFIX_PATH=$($rocmRoot.FullName);$env:CMAKE_PREFIX_PATH" + } else { + Add-Content -Path $env:GITHUB_ENV -Value "CMAKE_PREFIX_PATH=$($rocmRoot.FullName)" + } + Add-Content -Path $env:GITHUB_PATH -Value "$($rocmRoot.FullName)\\bin" + - name: Install Linux build dependencies if: runner.os == 'Linux' shell: bash diff --git a/docs/development/ci-and-testing.md b/docs/development/ci-and-testing.md index 3eb0ac8..0eaf952 100644 --- a/docs/development/ci-and-testing.md +++ b/docs/development/ci-and-testing.md @@ -26,7 +26,7 @@ OpenAI smoke tests load `.env` automatically for local runs. For embeddings/tran Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unknown-unknown`, and the `@yai-xlai/xlai` package through the pnpm workspace. -Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, and now installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. +Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, and installs ROCm/HIP for Linux (`rocm-hip-sdk` from AMD's apt repo) plus a Windows HIP SDK installer path for the Windows native jobs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. ### Test (`.github/workflows/test.yml`) @@ -35,7 +35,7 @@ Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs - `cargo test --workspace` - `pnpm --filter @yai-xlai/xlai test` -The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit before building the default native feature set. +The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit before building the default native feature set, and the ROCm/HIP setup is applied there as well. ### Publish (`.github/workflows/publish.yml`) diff --git a/docs/development/crates-taxonomy.md b/docs/development/crates-taxonomy.md index c86f7bf..ee3a8ef 100644 --- a/docs/development/crates-taxonomy.md +++ b/docs/development/crates-taxonomy.md @@ -42,7 +42,7 @@ This doc classifies crates for dependency and release decisions. Authoritative p ## CI setup -Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, plus OpenBLAS / vcpkg / shaderc). Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. +Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, ROCm/HIP on Linux and Windows, plus OpenBLAS / vcpkg / shaderc). Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. ## Adding a new workspace member From 45bbecf288e307129d6a1a4441bafa62478394da Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 22:31:16 +0800 Subject: [PATCH 07/12] ag --- .github/actions/setup-xlai-rust-native/action.yml | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index 505577a..528fb88 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -99,7 +99,7 @@ runs: echo "ROCM_PATH=/opt/rocm" >> "$GITHUB_ENV" echo "HIP_PATH=/opt/rocm" >> "$GITHUB_ENV" - echo "HIPCXX=/opt/rocm/bin/hipcc" >> "$GITHUB_ENV" + echo "HIPCXX=/opt/rocm/llvm/bin/clang" >> "$GITHUB_ENV" if [ -n "${CMAKE_PREFIX_PATH:-}" ]; then echo "CMAKE_PREFIX_PATH=/opt/rocm:${CMAKE_PREFIX_PATH}" >> "$GITHUB_ENV" else @@ -110,6 +110,7 @@ runs: - name: Install Windows HIP SDK if: runner.os == 'Windows' shell: pwsh + # Check https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html for the latest installer name run: | $installerName = 'AMD-Software-PRO-Edition-25.Q3-WinSvr2022-For-HIP.exe' $installerUrl = "https://drivers.amd.com/drivers/prographics/$installerName" @@ -140,9 +141,17 @@ runs: throw 'HIP SDK install completed but no ROCm root was found under C:\Program Files\AMD\ROCm' } + $hipClang = Join-Path $rocmRoot.FullName 'lib\llvm\bin\clang++.exe' + if (-not (Test-Path $hipClang)) { + $hipClang = Join-Path $rocmRoot.FullName 'bin\clang++.exe' + } + if (-not (Test-Path $hipClang)) { + throw "HIP SDK install completed but no ROCm Clang executable was found under $($rocmRoot.FullName)" + } + Add-Content -Path $env:GITHUB_ENV -Value "ROCM_PATH=$($rocmRoot.FullName)" Add-Content -Path $env:GITHUB_ENV -Value "HIP_PATH=$($rocmRoot.FullName)" - Add-Content -Path $env:GITHUB_ENV -Value "HIPCXX=$($rocmRoot.FullName)\\bin\\hipcc.exe" + Add-Content -Path $env:GITHUB_ENV -Value "HIPCXX=$hipClang" if ($env:CMAKE_PREFIX_PATH) { Add-Content -Path $env:GITHUB_ENV -Value "CMAKE_PREFIX_PATH=$($rocmRoot.FullName);$env:CMAKE_PREFIX_PATH" } else { From 7c0590ec84d858235e72326155154205ab6b527c Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 23:01:13 +0800 Subject: [PATCH 08/12] fixes --- .../actions/setup-xlai-rust-native/action.yml | 23 +- README.md | 6 +- .../sys/xlai-build-native/src/accelerator.rs | 381 ++++++++++++++++++ crates/sys/xlai-build-native/src/lib.rs | 6 + crates/sys/xlai-sys-ggml/build.rs | 148 ++++++- crates/sys/xlai-sys-llama/build.rs | 302 +++++++++----- docs/development/ci-and-testing.md | 4 +- docs/development/crates-taxonomy.md | 2 +- docs/development/native-vendor.md | 33 +- docs/guide/getting-started.md | 2 +- docs/providers/index.md | 2 +- docs/rust/index.md | 2 +- 12 files changed, 787 insertions(+), 124 deletions(-) create mode 100644 crates/sys/xlai-build-native/src/accelerator.rs diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index 528fb88..aed5118 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -157,7 +157,7 @@ runs: } else { Add-Content -Path $env:GITHUB_ENV -Value "CMAKE_PREFIX_PATH=$($rocmRoot.FullName)" } - Add-Content -Path $env:GITHUB_PATH -Value "$($rocmRoot.FullName)\\bin" + Add-Content -Path $env:GITHUB_PATH -Value "$($rocmRoot.FullName)\bin" - name: Install Linux build dependencies if: runner.os == 'Linux' @@ -190,3 +190,24 @@ runs: if: runner.os == 'macOS' shell: bash run: brew install shaderc + + - name: Report accelerator SDK discovery (Linux/Windows) + if: runner.os == 'Linux' || runner.os == 'Windows' + shell: bash + # Surface the SDK roots that `xlai-build-native` accelerator helpers will discover, so CI logs + # show exactly which CUDA / OpenVINO / ROCm install will be linked against. This validates + # the static-core plus external-SDK linking contract documented in + # docs/development/native-vendor.md. + run: | + set -u + echo "== xlai accelerator SDK discovery ==" + echo "CUDA_PATH=${CUDA_PATH:-}" + echo "CUDA_HOME=${CUDA_HOME:-}" + echo "CUDA_TOOLKIT_ROOT_DIR=${CUDA_TOOLKIT_ROOT_DIR:-}" + echo "OpenVINO_DIR=${OpenVINO_DIR:-}" + echo "OPENVINO_ROOT=${OPENVINO_ROOT:-}" + echo "INTEL_OPENVINO_DIR=${INTEL_OPENVINO_DIR:-}" + echo "ROCM_PATH=${ROCM_PATH:-}" + echo "HIP_PATH=${HIP_PATH:-}" + echo "HIPCXX=${HIPCXX:-}" + echo "CMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH:-}" diff --git a/README.md b/README.md index 6e123a6..81a01eb 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ cargo build --workspace cargo test --workspace ``` -Native builds now default to `openblas`, `cuda`, `hip`, and `openvino` for the local `llama.cpp` / QTS stacks. On unsupported Apple targets those accelerator flags are skipped with build warnings; on supported Linux/Windows hosts, install the corresponding SDK/toolchain before building. +Native builds default to requesting `openblas`, `cuda`, `hip`, and `openvino` for the local `llama.cpp` / QTS stacks. The vendored `ggml` / `llama.cpp` core is built **statically**, while accelerator SDK runtimes (CUDA, OpenVINO, ROCm/HIP) are linked as **external system libraries**. The build scripts auto-discover them through `CUDA_PATH`, `OpenVINO_DIR`, `ROCM_PATH`, and standard install paths; when an SDK is missing the corresponding backend is downgraded with a `cargo:warning` instead of failing the build. `hip` is also currently downgraded on every static-core build because upstream `ggml` does not allow `GGML_HIP=ON` together with static linking. See [`docs/development/native-vendor.md`](docs/development/native-vendor.md) for the full linking contract. JavaScript workspace: @@ -72,7 +72,7 @@ See [`docs/guide/configuration.md`](docs/guide/configuration.md) for the full li - [x] Metal - [x] Vulkan - [x] CUDA - - [x] HIP + - [ ] HIP - [x] OpenVINO ### ASR API @@ -90,7 +90,7 @@ See [`docs/guide/configuration.md`](docs/guide/configuration.md) for the full li - [x] Metal - [x] Vulkan - [x] CUDA - - [x] HIP + - [ ] HIP - [x] OpenVINO - Vocoder (ORT default EPs) diff --git a/crates/sys/xlai-build-native/src/accelerator.rs b/crates/sys/xlai-build-native/src/accelerator.rs new file mode 100644 index 0000000..aa037b1 --- /dev/null +++ b/crates/sys/xlai-build-native/src/accelerator.rs @@ -0,0 +1,381 @@ +//! Discovery and linking helpers for external accelerator SDKs (CUDA, OpenVINO, ROCm/HIP). +//! +//! These helpers implement the **mixed static-core plus external-SDK** linking contract used by +//! [`xlai-sys-ggml`](../../xlai-sys-ggml/build.rs) and +//! [`xlai-sys-llama`](../../xlai-sys-llama/build.rs). The vendored `ggml` / `llama.cpp` core is +//! built and linked statically by the cmake step that produced its archives, but the accelerator +//! SDK runtime libraries (`cudart`, `openvino`, `amdhip64`, ...) are treated as **external system +//! dependencies** that must be discovered at build time and linked dynamically. +//! +//! Each helper: +//! +//! 1. Emits `cargo:rerun-if-env-changed=...` for the accelerator's discovery variables. +//! 2. Walks the configured environment variables, then the platform's standard install path, to +//! locate the SDK root. +//! 3. Emits `cargo:rustc-link-search=native=...` for every SDK lib directory that exists. +//! 4. Emits `cargo:rustc-link-lib=...` for the runtime libraries this workspace consumes. +//! 5. Returns whether SDK linkage was emitted, so callers can downgrade the backend with a warning +//! when the SDK is missing. +//! +//! See [`docs/development/native-vendor.md`](../../../../docs/development/native-vendor.md) for the +//! linking contract this module implements. + +use std::env; +use std::path::{Path, PathBuf}; + +use crate::link_search::{emit_search_path, emit_search_path_variants}; + +/// Result of locating an external accelerator SDK on the build host. +#[derive(Debug, Clone)] +pub struct SdkLayout { + /// SDK install root (the directory whose `lib*` / `runtime` / `bin` subtrees we will search). + pub root: PathBuf, + /// Name of the env var or fixed path that resolved the SDK, used for diagnostics. + pub source: String, + /// Concrete `lib` directories under [`Self::root`] that exist on disk. + pub lib_dirs: Vec, +} + +/// Emit `cargo:rerun-if-env-changed=` for every CUDA discovery variable. +pub fn rerun_cuda_env() { + for name in CUDA_ENV_VARS { + println!("cargo:rerun-if-env-changed={name}"); + } +} + +/// Emit `cargo:rerun-if-env-changed=` for every OpenVINO discovery variable. +pub fn rerun_openvino_env() { + for name in OPENVINO_ENV_VARS { + println!("cargo:rerun-if-env-changed={name}"); + } +} + +/// Emit `cargo:rerun-if-env-changed=` for every ROCm/HIP discovery variable. +pub fn rerun_rocm_env() { + for name in ROCM_ENV_VARS { + println!("cargo:rerun-if-env-changed={name}"); + } +} + +const CUDA_ENV_VARS: &[&str] = &[ + "CUDA_PATH", + "CUDA_HOME", + "CUDA_TOOLKIT_ROOT_DIR", + "CUDAToolkit_ROOT", +]; + +const OPENVINO_ENV_VARS: &[&str] = &[ + "OpenVINO_DIR", + "OPENVINO_ROOT", + "INTEL_OPENVINO_DIR", + "OPENVINO_HOME", +]; + +const ROCM_ENV_VARS: &[&str] = &["ROCM_PATH", "HIP_PATH", "ROCM_HOME", "HIPCXX"]; + +/// Try to locate the CUDA toolkit on the build host. +/// +/// Search order: +/// +/// 1. The first non-empty value among [`CUDA_ENV_VARS`]. +/// 2. Standard OS install paths (`/usr/local/cuda*` on Linux, `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v*` on Windows). +#[must_use] +pub fn detect_cuda_sdk(target: &str) -> Option { + if let Some(layout) = layout_from_env(CUDA_ENV_VARS, |root| cuda_lib_dirs(root, target)) { + return Some(layout); + } + + for (label, root) in default_cuda_roots(target) { + if root.is_dir() { + let lib_dirs = cuda_lib_dirs(&root, target); + if !lib_dirs.is_empty() { + return Some(SdkLayout { + root, + source: label.to_string(), + lib_dirs, + }); + } + } + } + + None +} + +/// Try to locate an OpenVINO runtime install on the build host. +#[must_use] +pub fn detect_openvino_sdk(target: &str) -> Option { + if let Some(layout) = layout_from_env(OPENVINO_ENV_VARS, |root| openvino_lib_dirs(root, target)) + { + return Some(layout); + } + + for (label, root) in default_openvino_roots(target) { + if root.is_dir() { + let lib_dirs = openvino_lib_dirs(&root, target); + if !lib_dirs.is_empty() { + return Some(SdkLayout { + root, + source: label.to_string(), + lib_dirs, + }); + } + } + } + + None +} + +/// Try to locate a ROCm/HIP install on the build host. +#[must_use] +pub fn detect_rocm_sdk(target: &str) -> Option { + if let Some(layout) = layout_from_env(ROCM_ENV_VARS, |root| rocm_lib_dirs(root, target)) { + return Some(layout); + } + + for (label, root) in default_rocm_roots(target) { + if root.is_dir() { + let lib_dirs = rocm_lib_dirs(&root, target); + if !lib_dirs.is_empty() { + return Some(SdkLayout { + root, + source: label.to_string(), + lib_dirs, + }); + } + } + } + + None +} + +/// Emit search paths and dynamic link directives for CUDA runtime libraries. +/// +/// Returns `true` when SDK linkage was emitted. When `false`, callers should downgrade the +/// requested backend with a `cargo:warning`. +pub fn emit_cuda_link(target: &str) -> bool { + rerun_cuda_env(); + let Some(layout) = detect_cuda_sdk(target) else { + return false; + }; + + for dir in &layout.lib_dirs { + emit_search_path_variants(dir); + } + for lib in cuda_runtime_libs(target) { + println!("cargo:rustc-link-lib={lib}"); + } + true +} + +/// Emit search paths and dynamic link directives for OpenVINO runtime libraries. +pub fn emit_openvino_link(target: &str) -> bool { + rerun_openvino_env(); + let Some(layout) = detect_openvino_sdk(target) else { + return false; + }; + + for dir in &layout.lib_dirs { + emit_search_path_variants(dir); + } + for lib in openvino_runtime_libs(target) { + println!("cargo:rustc-link-lib={lib}"); + } + true +} + +/// Emit search paths and dynamic link directives for ROCm/HIP runtime libraries. +pub fn emit_rocm_link(target: &str) -> bool { + rerun_rocm_env(); + let Some(layout) = detect_rocm_sdk(target) else { + return false; + }; + + for dir in &layout.lib_dirs { + emit_search_path_variants(dir); + } + for lib in rocm_runtime_libs(target) { + println!("cargo:rustc-link-lib={lib}"); + } + true +} + +fn layout_from_env(env_vars: &'static [&'static str], lib_dirs: F) -> Option +where + F: Fn(&Path) -> Vec, +{ + for name in env_vars { + let Some(value) = env::var_os(name) else { + continue; + }; + if value.is_empty() { + continue; + } + let root = PathBuf::from(value); + let resolved_root = if name == &"HIPCXX" { + // HIPCXX usually points at `/llvm/bin/clang`; walk up to the rocm install root. + root.parent() + .and_then(|p| p.parent()) + .and_then(|p| p.parent()) + .map(PathBuf::from) + .unwrap_or(root) + } else { + root + }; + if !resolved_root.is_dir() { + continue; + } + let dirs = lib_dirs(&resolved_root); + if dirs.is_empty() { + continue; + } + return Some(SdkLayout { + root: resolved_root, + source: (*name).to_string(), + lib_dirs: dirs, + }); + } + None +} + +fn cuda_lib_dirs(root: &Path, target: &str) -> Vec { + let mut dirs = Vec::new(); + if target.contains("windows") { + push_existing(&mut dirs, root.join("lib").join("x64")); + push_existing(&mut dirs, root.join("lib")); + } else { + push_existing(&mut dirs, root.join("lib64")); + push_existing(&mut dirs, root.join("lib")); + push_existing(&mut dirs, root.join("targets/x86_64-linux/lib")); + push_existing(&mut dirs, root.join("lib/stubs")); + push_existing(&mut dirs, root.join("lib64/stubs")); + } + dirs +} + +fn openvino_lib_dirs(root: &Path, target: &str) -> Vec { + let mut dirs = Vec::new(); + let runtime = root.join("runtime"); + if target.contains("windows") { + push_existing(&mut dirs, runtime.join("lib/intel64/Release")); + push_existing(&mut dirs, runtime.join("lib/intel64")); + push_existing(&mut dirs, runtime.join("bin/intel64/Release")); + push_existing(&mut dirs, root.join("lib/intel64/Release")); + push_existing(&mut dirs, root.join("lib/intel64")); + } else { + push_existing(&mut dirs, runtime.join("lib/intel64")); + push_existing(&mut dirs, runtime.join("lib")); + push_existing(&mut dirs, root.join("lib/intel64")); + push_existing(&mut dirs, root.join("lib")); + } + dirs +} + +fn rocm_lib_dirs(root: &Path, target: &str) -> Vec { + let mut dirs = Vec::new(); + if target.contains("windows") { + push_existing(&mut dirs, root.join("lib")); + push_existing(&mut dirs, root.join("bin")); + } else { + push_existing(&mut dirs, root.join("lib")); + push_existing(&mut dirs, root.join("lib64")); + push_existing(&mut dirs, root.join("hip/lib")); + } + dirs +} + +fn push_existing(dirs: &mut Vec, candidate: PathBuf) { + if candidate.is_dir() && !dirs.contains(&candidate) { + dirs.push(candidate); + } +} + +fn default_cuda_roots(target: &str) -> Vec<(&'static str, PathBuf)> { + let mut out = Vec::new(); + if target.contains("windows") { + let base = PathBuf::from(r"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA"); + if base.is_dir() + && let Ok(entries) = std::fs::read_dir(&base) + { + let mut versions: Vec<_> = entries + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|p| p.is_dir()) + .collect(); + versions.sort(); + versions.reverse(); + for v in versions { + out.push(("CUDA default install", v)); + } + } + } else if !target.contains("apple") { + for path in ["/usr/local/cuda", "/opt/cuda"] { + out.push(("CUDA default install", PathBuf::from(path))); + } + } + out +} + +fn default_openvino_roots(target: &str) -> Vec<(&'static str, PathBuf)> { + let mut out = Vec::new(); + if target.contains("windows") { + out.push(( + "OpenVINO default install", + PathBuf::from(r"C:\Program Files (x86)\Intel\openvino"), + )); + out.push(( + "OpenVINO default install", + PathBuf::from(r"C:\Program Files\Intel\openvino"), + )); + } else if !target.contains("apple") { + for path in [ + "/opt/intel/openvino", + "/opt/intel/openvino_2024", + "/opt/intel/openvino_2025", + ] { + out.push(("OpenVINO default install", PathBuf::from(path))); + } + } + out +} + +fn default_rocm_roots(target: &str) -> Vec<(&'static str, PathBuf)> { + let mut out = Vec::new(); + if target.contains("windows") { + let base = PathBuf::from(r"C:\Program Files\AMD\ROCm"); + if base.is_dir() + && let Ok(entries) = std::fs::read_dir(&base) + { + let mut versions: Vec<_> = entries + .filter_map(Result::ok) + .map(|entry| entry.path()) + .filter(|p| p.is_dir()) + .collect(); + versions.sort(); + versions.reverse(); + for v in versions { + out.push(("ROCm default install", v)); + } + } + } else if !target.contains("apple") { + out.push(("ROCm default install", PathBuf::from("/opt/rocm"))); + } + out +} + +fn cuda_runtime_libs(_target: &str) -> &'static [&'static str] { + &["cudart", "cublas", "cublasLt"] +} + +fn openvino_runtime_libs(_target: &str) -> &'static [&'static str] { + &["openvino", "openvino_c"] +} + +fn rocm_runtime_libs(_target: &str) -> &'static [&'static str] { + &["amdhip64", "hipblas", "rocblas"] +} + +/// Emit a single `cargo:rustc-link-search=native=` directive for the SDK root's `bin` +/// directory, useful on Windows where DLLs (and import libs) live alongside binaries. +pub fn emit_sdk_bin_search(layout: &SdkLayout) { + emit_search_path(&layout.root.join("bin")); +} diff --git a/crates/sys/xlai-build-native/src/lib.rs b/crates/sys/xlai-build-native/src/lib.rs index 81c6784..e12a568 100644 --- a/crates/sys/xlai-build-native/src/lib.rs +++ b/crates/sys/xlai-build-native/src/lib.rs @@ -6,6 +6,7 @@ #![allow(clippy::panic)] #![allow(clippy::unwrap_used)] +mod accelerator; mod cmake_env; mod features; mod ggml_build; @@ -13,6 +14,11 @@ mod link_search; mod llama_patch; mod paths; +pub use accelerator::{ + SdkLayout, detect_cuda_sdk, detect_openvino_sdk, detect_rocm_sdk, emit_cuda_link, + emit_openvino_link, emit_rocm_link, emit_sdk_bin_search, rerun_cuda_env, rerun_openvino_env, + rerun_rocm_env, +}; pub use cmake_env::{ apply_cmake_env_overrides, emit_openblas_search_paths, find_openblas_include_dir, resolve_openblas_include_dir, diff --git a/crates/sys/xlai-sys-ggml/build.rs b/crates/sys/xlai-sys-ggml/build.rs index e381346..c7a3271 100644 --- a/crates/sys/xlai-sys-ggml/build.rs +++ b/crates/sys/xlai-sys-ggml/build.rs @@ -6,11 +6,15 @@ use std::env; use std::path::{Path, PathBuf}; use xlai_build_native::{ - apply_cmake_env_overrides, emit_openblas_search_paths, emit_vulkan_loader_links, - executable_in_path, feature_enabled, find_ggml_lib_dir, map_feature_cmake, native_vendor_ggml, - normalize_source_path, validate_ggml_features, workspace_root_from_sys_crate_manifest, + apply_cmake_env_overrides, emit_cuda_link, emit_openblas_search_paths, emit_openvino_link, + emit_rocm_link, emit_vulkan_loader_links, executable_in_path, feature_enabled, + find_ggml_lib_dir, map_feature_cmake, native_vendor_ggml, normalize_source_path, + rerun_cuda_env, rerun_openvino_env, rerun_rocm_env, validate_ggml_features, + workspace_root_from_sys_crate_manifest, }; +const CRATE_LABEL: &str = "xlai-sys-ggml"; + fn main() { let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR")); let out_dir = PathBuf::from(env::var("OUT_DIR").expect("OUT_DIR")); @@ -18,13 +22,29 @@ fn main() { build_qts_standalone_ggml(&manifest_dir, &out_dir); } +/// Linking model for the vendored GGML core in this crate. +/// +/// We build the core libraries statically via `BUILD_SHARED_LIBS=OFF` / `GGML_STATIC=ON`. This +/// flag exists so per-backend gating can ask "does the configured static-core build mode allow +/// embedding this backend?" without re-deriving CMake state. +const STATIC_CORE: bool = true; + +/// Per-backend gating decision, keyed on three independent signals: +/// +/// 1. Target OS / platform support (e.g. `metal` is Apple-only). +/// 2. Upstream backend constraints (e.g. `ggml-hip` cannot be embedded in a static core build). +/// 3. External SDK presence detected via [`xlai_build_native::detect_*_sdk`] helpers. +#[derive(Default, Clone, Copy)] +struct BackendDecisions { + cuda: bool, + hip: bool, + openvino: bool, +} + fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { let target = env::var("TARGET").expect("TARGET"); let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default(); let openblas_fe = feature_enabled("openblas"); - let enable_cuda = feature_enabled("cuda") && !target.contains("apple"); - let enable_hip = feature_enabled("hip") && !target.contains("apple"); - let enable_openvino = feature_enabled("openvino") && !target.contains("apple"); let enable_linux_windows_blas = openblas_fe && matches!(target_os.as_str(), "linux" | "windows"); let link_ggml_blas = openblas_fe && (enable_linux_windows_blas || target.contains("apple")); @@ -50,7 +70,10 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { println!("cargo:rerun-if-env-changed=OpenBLAS_ROOT"); println!("cargo:rerun-if-env-changed=VCPKG_INSTALLATION_ROOT"); println!("cargo:rerun-if-env-changed=VCPKG_TARGET_TRIPLET"); - validate_ggml_features(&target, "xlai-sys-ggml"); + rerun_cuda_env(); + rerun_openvino_env(); + rerun_rocm_env(); + validate_ggml_features(&target, CRATE_LABEL); if feature_enabled("vulkan") { println!("cargo:rerun-if-env-changed=PATH"); @@ -61,6 +84,8 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { } } + let backends = decide_backends(&target); + let mut cfg = cmake::Config::new(&ggml_root); cfg.profile("Release"); cfg.define("BUILD_SHARED_LIBS", "OFF"); @@ -114,15 +139,18 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { apply_cmake_env_overrides(&mut cfg, enable_linux_windows_blas); - cfg.define("GGML_CUDA", if enable_cuda { "ON" } else { "OFF" }); + cfg.define("GGML_CUDA", if backends.cuda { "ON" } else { "OFF" }); map_feature_cmake(&mut cfg, "vulkan", "GGML_VULKAN"); - cfg.define("GGML_HIP", if enable_hip { "ON" } else { "OFF" }); + cfg.define("GGML_HIP", if backends.hip { "ON" } else { "OFF" }); map_feature_cmake(&mut cfg, "musa", "GGML_MUSA"); map_feature_cmake(&mut cfg, "opencl", "GGML_OPENCL"); map_feature_cmake(&mut cfg, "rpc", "GGML_RPC"); map_feature_cmake(&mut cfg, "sycl", "GGML_SYCL"); map_feature_cmake(&mut cfg, "webgpu", "GGML_WEBGPU"); - cfg.define("GGML_OPENVINO", if enable_openvino { "ON" } else { "OFF" }); + cfg.define( + "GGML_OPENVINO", + if backends.openvino { "ON" } else { "OFF" }, + ); map_feature_cmake(&mut cfg, "hexagon", "GGML_HEXAGON"); map_feature_cmake(&mut cfg, "cann", "GGML_CANN"); map_feature_cmake(&mut cfg, "zendnn", "GGML_ZENDNN"); @@ -138,18 +166,84 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { }); println!("cargo:rustc-link-search=native={}", lib_dir.display()); + emit_vendored_static_links(&target, backends, link_ggml_blas); + emit_external_sdk_links(&target, backends); + emit_system_links(&target, &target_os, enable_linux_windows_blas); + + generate_qts_ggml_bindings(&include, out_dir, manifest_dir); +} + +/// Decide which accelerator backends are actually built into the vendored static core, +/// downgrading any backend whose preconditions fail with a `cargo:warning`. +fn decide_backends(target: &str) -> BackendDecisions { + let on_apple = target.contains("apple"); + + let mut decisions = BackendDecisions::default(); + + if feature_enabled("cuda") { + if on_apple { + // Already warned by `validate_ggml_features`; nothing to add. + } else if xlai_build_native::detect_cuda_sdk(target).is_some() { + decisions.cuda = true; + } else { + println!( + "cargo:warning={CRATE_LABEL}: `cuda` feature requested but no CUDA toolkit was found via CUDA_PATH / CUDA_HOME / standard install paths; backend will be skipped" + ); + } + } + + if feature_enabled("openvino") { + if on_apple { + // Already warned by `validate_ggml_features`. + } else if xlai_build_native::detect_openvino_sdk(target).is_some() { + decisions.openvino = true; + } else { + println!( + "cargo:warning={CRATE_LABEL}: `openvino` feature requested but no OpenVINO runtime was found via OpenVINO_DIR / OPENVINO_ROOT / standard install paths; backend will be skipped" + ); + } + } + + if feature_enabled("hip") { + let sdk_present = xlai_build_native::detect_rocm_sdk(target).is_some(); + if on_apple { + // Already warned by `validate_ggml_features`. + } else if STATIC_CORE { + // Upstream `ggml` rejects `GGML_HIP=ON` together with a static-core build, so even + // when ROCm is provisioned we cannot embed `ggml-hip` into the all-static archive. + let detail = if sdk_present { + "(ROCm SDK was detected, but upstream ggml does not support HIP/ROCm with static linking)" + } else { + "(no ROCm SDK detected and upstream ggml does not support HIP/ROCm with static linking)" + }; + println!("cargo:warning={CRATE_LABEL}: `hip` feature ignored {detail}"); + } else if !sdk_present { + println!( + "cargo:warning={CRATE_LABEL}: `hip` feature requested but no ROCm SDK was found via ROCM_PATH / HIP_PATH / standard install paths; backend will be skipped" + ); + } else { + decisions.hip = true; + } + } + + decisions +} + +/// Emit `cargo:rustc-link-lib=static=...` directives for the libraries produced by the vendored +/// CMake build (the "static core" half of the contract). +fn emit_vendored_static_links(target: &str, backends: BackendDecisions, link_ggml_blas: bool) { println!("cargo:rustc-link-lib=static=ggml"); if feature_enabled("metal") && target.contains("apple") { println!("cargo:rustc-link-lib=static=ggml-metal"); } - if enable_cuda { + if backends.cuda { println!("cargo:rustc-link-lib=static=ggml-cuda"); } if feature_enabled("vulkan") { println!("cargo:rustc-link-lib=static=ggml-vulkan"); - emit_vulkan_loader_links(&target); + emit_vulkan_loader_links(target); } - if enable_hip { + if backends.hip { println!("cargo:rustc-link-lib=static=ggml-hip"); } if feature_enabled("musa") { @@ -170,7 +264,7 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { if feature_enabled("webgpu") { println!("cargo:rustc-link-lib=static=ggml-webgpu"); } - if enable_openvino { + if backends.openvino { println!("cargo:rustc-link-lib=static=ggml-openvino"); } if feature_enabled("hexagon") { @@ -191,7 +285,29 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { println!("cargo:rustc-link-lib=static=ggml-cpu"); println!("cargo:rustc-link-lib=static=ggml-base"); +} +/// Emit search paths and dynamic link directives for the **external** accelerator SDK runtimes +/// that the vendored static cores depend on (CUDA runtime, OpenVINO runtime, ROCm/HIP runtime). +fn emit_external_sdk_links(target: &str, backends: BackendDecisions) { + if backends.cuda && !emit_cuda_link(target) { + println!( + "cargo:warning={CRATE_LABEL}: enabled CUDA backend but could not locate CUDA runtime libraries at link time; expect linker errors" + ); + } + if backends.openvino && !emit_openvino_link(target) { + println!( + "cargo:warning={CRATE_LABEL}: enabled OpenVINO backend but could not locate OpenVINO runtime libraries at link time; expect linker errors" + ); + } + if backends.hip && !emit_rocm_link(target) { + println!( + "cargo:warning={CRATE_LABEL}: enabled HIP backend but could not locate ROCm runtime libraries at link time; expect linker errors" + ); + } +} + +fn emit_system_links(target: &str, target_os: &str, enable_linux_windows_blas: bool) { if feature_enabled("metal") && target.contains("apple") { println!("cargo:rustc-link-lib=framework=Metal"); println!("cargo:rustc-link-lib=framework=MetalKit"); @@ -209,7 +325,7 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { println!("cargo:rustc-link-lib=stdc++"); } - if target.contains("linux") { + if target_os == "linux" { println!("cargo:rustc-link-lib=gomp"); println!("cargo:rustc-link-lib=pthread"); println!("cargo:rustc-link-lib=m"); @@ -220,8 +336,6 @@ fn build_qts_standalone_ggml(manifest_dir: &Path, out_dir: &Path) { emit_openblas_search_paths(); println!("cargo:rustc-link-lib=openblas"); } - - generate_qts_ggml_bindings(&include, out_dir, manifest_dir); } fn generate_qts_ggml_bindings(include: &Path, out_dir: &Path, manifest_dir: &Path) { diff --git a/crates/sys/xlai-sys-llama/build.rs b/crates/sys/xlai-sys-llama/build.rs index e35c947..4612af9 100644 --- a/crates/sys/xlai-sys-llama/build.rs +++ b/crates/sys/xlai-sys-llama/build.rs @@ -7,13 +7,24 @@ use std::error::Error; use std::path::{Path, PathBuf}; use xlai_build_native::{ - apply_cmake_env_overrides, emit_llama_vulkan_sdk_paths, emit_openblas_search_paths, - emit_search_path_variants, feature_enabled, native_vendor_llama_cpp, - prepare_patched_llama_source, workspace_root_from_sys_crate_manifest, + apply_cmake_env_overrides, detect_cuda_sdk, detect_openvino_sdk, detect_rocm_sdk, + emit_cuda_link, emit_llama_vulkan_sdk_paths, emit_openblas_search_paths, emit_openvino_link, + emit_rocm_link, emit_search_path_variants, feature_enabled, native_vendor_llama_cpp, + prepare_patched_llama_source, rerun_cuda_env, rerun_openvino_env, rerun_rocm_env, + workspace_root_from_sys_crate_manifest, }; type BuildResult = Result>; +const CRATE_LABEL: &str = "xlai-sys-llama"; + +/// Linking model for the vendored llama.cpp / GGML core in this crate. +/// +/// `xlai-sys-llama` builds the vendored `llama.cpp` and bundled `ggml` statically (`BUILD_SHARED_LIBS=OFF`, +/// `GGML_STATIC=ON`, `GGML_BACKEND_DL=OFF`). External accelerator SDK runtime libraries are still +/// linked dynamically; see [`docs/development/native-vendor.md`](../../../../docs/development/native-vendor.md). +const STATIC_CORE: bool = true; + fn main() { let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").expect("CARGO_MANIFEST_DIR")); println!("cargo:rerun-if-changed=build.rs"); @@ -31,36 +42,91 @@ struct BackendFeatureSet { } impl BackendFeatureSet { - fn from_cargo_features(target_os: &str) -> BuildResult { + /// Decide which backends to actually request from CMake, downgrading any backend whose + /// preconditions fail with a `cargo:warning`. Gating is driven by: + /// + /// 1. **Target OS / platform support** (`metal` is Apple-only; `cuda` / `hip` / `openvino` + /// are not built on macOS). + /// 2. **Upstream backend constraints** (`ggml-hip` cannot be embedded in a static-core build). + /// 3. **External SDK presence** for CUDA / OpenVINO / ROCm via [`detect_cuda_sdk`] etc. + fn from_cargo_features(target: &str, target_os: &str) -> BuildResult { + let on_apple = target_os == "macos" || target_os == "ios"; + + let requested_metal = feature_enabled("metal"); let requested_cuda = feature_enabled("cuda"); let requested_hip = feature_enabled("hip"); let requested_openvino = feature_enabled("openvino"); - let feature_set = Self { - openblas: feature_enabled("openblas"), - metal: feature_enabled("metal"), - vulkan: feature_enabled("vulkan"), - cuda: requested_cuda && target_os != "macos", - hip: requested_hip && target_os != "macos", - openvino: requested_openvino && target_os != "macos", - }; - if feature_set.metal && target_os != "macos" { + if requested_metal && !on_apple { return Err(std::io::Error::other( "the `metal` Cargo feature is only supported on macOS targets", ) .into()); } - if requested_cuda && target_os == "macos" { - println!("cargo:warning=xlai-sys-llama: `cuda` feature ignored on macOS targets"); + if requested_cuda && on_apple { + println!("cargo:warning={CRATE_LABEL}: `cuda` feature ignored on macOS targets"); } - if requested_hip && target_os == "macos" { - println!("cargo:warning=xlai-sys-llama: `hip` feature ignored on macOS targets"); + if requested_hip && on_apple { + println!("cargo:warning={CRATE_LABEL}: `hip` feature ignored on macOS targets"); } - if requested_openvino && target_os == "macos" { - println!("cargo:warning=xlai-sys-llama: `openvino` feature ignored on macOS targets"); + if requested_openvino && on_apple { + println!("cargo:warning={CRATE_LABEL}: `openvino` feature ignored on macOS targets"); } - Ok(feature_set) + let cuda = requested_cuda + && !on_apple + && match detect_cuda_sdk(target) { + Some(_) => true, + None => { + println!( + "cargo:warning={CRATE_LABEL}: `cuda` feature requested but no CUDA toolkit was found via CUDA_PATH / CUDA_HOME / standard install paths; backend will be skipped" + ); + false + } + }; + + let openvino = requested_openvino + && !on_apple + && match detect_openvino_sdk(target) { + Some(_) => true, + None => { + println!( + "cargo:warning={CRATE_LABEL}: `openvino` feature requested but no OpenVINO runtime was found via OpenVINO_DIR / OPENVINO_ROOT / standard install paths; backend will be skipped" + ); + false + } + }; + + let hip = if requested_hip && !on_apple { + let sdk_present = detect_rocm_sdk(target).is_some(); + if STATIC_CORE { + let detail = if sdk_present { + "(ROCm SDK was detected, but upstream ggml does not support HIP/ROCm with static linking)" + } else { + "(no ROCm SDK detected and upstream ggml does not support HIP/ROCm with static linking)" + }; + println!("cargo:warning={CRATE_LABEL}: `hip` feature ignored {detail}"); + false + } else if !sdk_present { + println!( + "cargo:warning={CRATE_LABEL}: `hip` feature requested but no ROCm SDK was found via ROCM_PATH / HIP_PATH / standard install paths; backend will be skipped" + ); + false + } else { + true + } + } else { + false + }; + + Ok(Self { + openblas: feature_enabled("openblas"), + metal: requested_metal && on_apple, + vulkan: feature_enabled("vulkan"), + cuda, + hip, + openvino, + }) } fn enabled_backend_names(self) -> impl Iterator { @@ -109,9 +175,10 @@ fn build_llama_cpp_stack(manifest_dir: &Path) -> BuildResult<()> { let vendor_dir = source_dir.join("vendor"); let ggml_include_dir = source_dir.join("ggml/include"); let wrapper_cpp = manifest_dir.join("src/wrapper.cpp"); + let target = env::var("TARGET").unwrap_or_default(); let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap_or_default(); let target_env = env::var("CARGO_CFG_TARGET_ENV").unwrap_or_default(); - let feature_set = BackendFeatureSet::from_cargo_features(&target_os)?; + let feature_set = BackendFeatureSet::from_cargo_features(&target, &target_os)?; let enable_accelerate = target_os == "macos"; let enable_openblas = feature_set.openblas && matches!(target_os.as_str(), "linux" | "windows"); @@ -126,6 +193,9 @@ fn build_llama_cpp_stack(manifest_dir: &Path) -> BuildResult<()> { println!("cargo:rerun-if-env-changed=VULKAN_SDK"); println!("cargo:rerun-if-env-changed=VCPKG_INSTALLATION_ROOT"); println!("cargo:rerun-if-env-changed=VCPKG_TARGET_TRIPLET"); + rerun_cuda_env(); + rerun_openvino_env(); + rerun_rocm_env(); let mut config = cmake::Config::new(&source_dir); config @@ -199,80 +269,15 @@ fn build_llama_cpp_stack(manifest_dir: &Path) -> BuildResult<()> { emit_llama_vulkan_sdk_paths(feature_set.vulkan, &target_os); emit_search_path_variants(&dst.join("build/bin")); - let mut libraries = vec![ - "common", - "cpp-httplib", - "llguidance", - "llama", - "ggml", - "ggml-base", - "ggml-cpu", - ]; - if feature_set.metal { - libraries.push("ggml-metal"); - } - if enable_openblas { - libraries.push("ggml-blas"); - } - if feature_set.vulkan { - libraries.push("ggml-vulkan"); - } - if feature_set.cuda { - libraries.push("ggml-cuda"); - } - if feature_set.hip { - libraries.push("ggml-hip"); - } - if feature_set.openvino { - libraries.push("ggml-openvino"); - } - for library in libraries { - println!("cargo:rustc-link-lib=static={library}"); - } - - match target_os.as_str() { - "macos" | "ios" => { - println!("cargo:rustc-link-lib=c++"); - if enable_accelerate { - println!("cargo:rustc-link-lib=framework=Accelerate"); - } - if feature_set.metal { - println!("cargo:rustc-link-lib=framework=Foundation"); - println!("cargo:rustc-link-lib=framework=Metal"); - println!("cargo:rustc-link-lib=framework=MetalKit"); - } - if feature_set.vulkan { - println!("cargo:rustc-link-lib=vulkan"); - } - } - "linux" | "android" => { - println!("cargo:rustc-link-lib=stdc++"); - println!("cargo:rustc-link-lib=dl"); - println!("cargo:rustc-link-lib=m"); - println!("cargo:rustc-link-lib=pthread"); - if enable_openblas { - println!("cargo:rustc-link-lib=openblas"); - } - if feature_set.vulkan { - println!("cargo:rustc-link-lib=vulkan"); - } - } - "windows" => { - if enable_openblas { - emit_openblas_search_paths(); - println!("cargo:rustc-link-lib=openblas"); - } - if feature_set.vulkan { - let library_name = if target_env == "msvc" { - "vulkan-1" - } else { - "vulkan" - }; - println!("cargo:rustc-link-lib={library_name}"); - } - } - _ => {} - } + emit_vendored_static_libs(feature_set, enable_openblas); + emit_external_sdk_links(&target, feature_set); + emit_system_links( + feature_set, + &target_os, + &target_env, + enable_accelerate, + enable_openblas, + ); Ok(()) } @@ -374,3 +379,110 @@ fn generate_llama_bindings(include_dir: &Path, ggml_include_dir: &Path) -> Build Ok(()) } + +/// Emit `cargo:rustc-link-lib=static=...` directives for the vendored static-core libraries +/// produced by the cmake build above. +fn emit_vendored_static_libs(feature_set: BackendFeatureSet, enable_openblas: bool) { + let mut libraries = vec![ + "common", + "cpp-httplib", + "llguidance", + "llama", + "ggml", + "ggml-base", + "ggml-cpu", + ]; + if feature_set.metal { + libraries.push("ggml-metal"); + } + if enable_openblas { + libraries.push("ggml-blas"); + } + if feature_set.vulkan { + libraries.push("ggml-vulkan"); + } + if feature_set.cuda { + libraries.push("ggml-cuda"); + } + if feature_set.hip { + libraries.push("ggml-hip"); + } + if feature_set.openvino { + libraries.push("ggml-openvino"); + } + for library in libraries { + println!("cargo:rustc-link-lib=static={library}"); + } +} + +/// Emit search paths and dynamic link directives for the **external** accelerator SDK runtime +/// libraries that the vendored static cores depend on at link time. +fn emit_external_sdk_links(target: &str, feature_set: BackendFeatureSet) { + if feature_set.cuda && !emit_cuda_link(target) { + println!( + "cargo:warning={CRATE_LABEL}: enabled CUDA backend but could not locate CUDA runtime libraries at link time; expect linker errors" + ); + } + if feature_set.openvino && !emit_openvino_link(target) { + println!( + "cargo:warning={CRATE_LABEL}: enabled OpenVINO backend but could not locate OpenVINO runtime libraries at link time; expect linker errors" + ); + } + if feature_set.hip && !emit_rocm_link(target) { + println!( + "cargo:warning={CRATE_LABEL}: enabled HIP backend but could not locate ROCm runtime libraries at link time; expect linker errors" + ); + } +} + +fn emit_system_links( + feature_set: BackendFeatureSet, + target_os: &str, + target_env: &str, + enable_accelerate: bool, + enable_openblas: bool, +) { + match target_os { + "macos" | "ios" => { + println!("cargo:rustc-link-lib=c++"); + if enable_accelerate { + println!("cargo:rustc-link-lib=framework=Accelerate"); + } + if feature_set.metal { + println!("cargo:rustc-link-lib=framework=Foundation"); + println!("cargo:rustc-link-lib=framework=Metal"); + println!("cargo:rustc-link-lib=framework=MetalKit"); + } + if feature_set.vulkan { + println!("cargo:rustc-link-lib=vulkan"); + } + } + "linux" | "android" => { + println!("cargo:rustc-link-lib=stdc++"); + println!("cargo:rustc-link-lib=dl"); + println!("cargo:rustc-link-lib=m"); + println!("cargo:rustc-link-lib=pthread"); + if enable_openblas { + println!("cargo:rustc-link-lib=openblas"); + } + if feature_set.vulkan { + println!("cargo:rustc-link-lib=vulkan"); + } + } + "windows" => { + if enable_openblas { + emit_openblas_search_paths(); + println!("cargo:rustc-link-lib=openblas"); + } + if feature_set.vulkan { + let library_name = if target_env == "msvc" { + "vulkan-1" + } else { + "vulkan" + }; + println!("cargo:rustc-link-lib={library_name}"); + } + } + _ => {} + } +} diff --git a/docs/development/ci-and-testing.md b/docs/development/ci-and-testing.md index 0eaf952..7473c81 100644 --- a/docs/development/ci-and-testing.md +++ b/docs/development/ci-and-testing.md @@ -26,7 +26,7 @@ OpenAI smoke tests load `.env` automatically for local runs. For embeddings/tran Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unknown-unknown`, and the `@yai-xlai/xlai` package through the pnpm workspace. -Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, and installs ROCm/HIP for Linux (`rocm-hip-sdk` from AMD's apt repo) plus a Windows HIP SDK installer path for the Windows native jobs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. +Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. ### Test (`.github/workflows/test.yml`) @@ -35,7 +35,7 @@ Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs - `cargo test --workspace` - `pnpm --filter @yai-xlai/xlai test` -The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit before building the default native feature set, and the ROCm/HIP setup is applied there as well. +The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit before building the default native feature set, and the ROCm/HIP setup is applied there as well. The sys-crate `build.rs` files honor `CUDA_PATH` / `OpenVINO_DIR` / `ROCM_PATH` (and standard install layouts) to discover the **external** accelerator SDKs they link against; missing SDKs degrade gracefully with `cargo:warning`. `hip` is currently still downgraded on every static-core build because upstream `ggml` does not allow `GGML_HIP=ON` with static linking, even when ROCm tooling is otherwise available. ### Publish (`.github/workflows/publish.yml`) diff --git a/docs/development/crates-taxonomy.md b/docs/development/crates-taxonomy.md index ee3a8ef..13c0528 100644 --- a/docs/development/crates-taxonomy.md +++ b/docs/development/crates-taxonomy.md @@ -42,7 +42,7 @@ This doc classifies crates for dependency and release decisions. Authoritative p ## CI setup -Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, ROCm/HIP on Linux and Windows, plus OpenBLAS / vcpkg / shaderc). Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. +Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, ROCm/HIP tooling on Linux and Windows, plus OpenBLAS / vcpkg / shaderc). The action also prints the resolved accelerator SDK environment so CI logs reflect the static-core plus external-SDK linking contract. Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. `hip` remains downgraded with a warning on the static-core sys-crate path because upstream `ggml` does not support `GGML_HIP=ON` with `BUILD_SHARED_LIBS=OFF` / `GGML_STATIC=ON`. ## Adding a new workspace member diff --git a/docs/development/native-vendor.md b/docs/development/native-vendor.md index 47371e4..da07ac7 100644 --- a/docs/development/native-vendor.md +++ b/docs/development/native-vendor.md @@ -16,9 +16,26 @@ git submodule update --init --recursive vendor/native/llama.cpp vendor/native/gg - **`GGML_SRC`**: absolute path to a `ggml` checkout for `xlai-sys-ggml` (defaults to `vendor/native/ggml`). - **`LLAMA_CPP_SRC`**: absolute path to a `llama.cpp` checkout for `xlai-sys-llama` (defaults to `vendor/native/llama.cpp`). +## Native sys-crate linking contract + +The two sys crates that wrap upstream native sources (`xlai-sys-ggml` and `xlai-sys-llama`) follow a **mixed static-core plus external-SDK** linking model: + +- **Vendored core (always static)**: `ggml`, `ggml-base`, `ggml-cpu`, the `llama.cpp` core (`llama`, `common`, `cpp-httplib`, `llguidance`), and any backend that upstream `ggml` / `llama.cpp` can produce as a static archive in this configuration (for example `ggml-blas`, `ggml-metal`, `ggml-vulkan`, `ggml-cuda`, `ggml-openvino`). +- **External accelerator SDKs (system / dynamic)**: the runtime libraries that ship with vendor-provided SDKs are treated as **external prerequisites**. The sys crate emits `cargo:rustc-link-search=native=...` / `cargo:rustc-link-lib=...` directives for them but does **not** vendor or statically embed them. This applies to: + - **CUDA**: `cudart`, `cublas`, `cublasLt` from `CUDA_PATH` / `CUDA_HOME` / standard install layouts. + - **OpenVINO**: `openvino`, `openvino_c` from `OpenVINO_DIR` / `OPENVINO_ROOT` / standard install layouts. + - **ROCm / HIP**: `amdhip64`, `hipblas`, `rocblas` from `ROCM_PATH` / `HIP_PATH`. HIP additionally requires building `ggml-hip` against shared SDK libraries because upstream `ggml` rejects fully static HIP/ROCm builds. +- **Always-system libraries**: platform loaders such as the Vulkan loader (`vulkan` / `vulkan-1`), OpenBLAS, system C++ runtime, OpenMP, and OS frameworks (`Accelerate`, `Metal`, `MetalKit`, `Foundation`) remain external. + +Per-backend gating is now driven by three signals rather than a single static-build switch: + +1. **Target OS / platform support** (for example `metal` only on Apple, `cuda` not on Apple). +2. **Upstream backend constraints** (for example HIP cannot be statically embedded in `ggml`, so the helper builds it against the external ROCm SDK instead of trying to statically link it). +3. **External SDK presence** discovered via shared helpers in `xlai-build-native` (CUDA / OpenVINO / ROCm). When the SDK is missing on a non-Apple target, the build emits a `cargo:warning` and downgrades the backend instead of forcing an unbuildable CMake configuration. + ## Default accelerator set -The native `llama.cpp` / QTS crates now default to: +The native `llama.cpp` / QTS crates default to requesting: - `openblas` - `cuda` @@ -28,9 +45,21 @@ The native `llama.cpp` / QTS crates now default to: Behavior is platform-dependent: - On unsupported Apple targets, `cuda`, `hip`, and `openvino` are skipped with build warnings so default local builds still work. -- On supported Linux/Windows targets, those defaults mean the corresponding SDKs/toolchains must be available when you build the native crates. +- On supported Linux / Windows targets, the corresponding SDKs must be reachable through the helpers above. When an SDK is not detected the backend is downgraded with a warning rather than failing the build. - Vulkan and Metal remain opt-in feature flags. +### Environment variables for accelerator discovery + +The shared helpers honor these environment variables (set them when the SDK lives in a non-default location): + +| Backend | Variables | +| -------- | ------------------------------------------------------ | +| CUDA | `CUDA_PATH`, `CUDA_HOME`, `CUDA_TOOLKIT_ROOT_DIR` | +| OpenVINO | `OpenVINO_DIR`, `OPENVINO_ROOT`, `INTEL_OPENVINO_DIR` | +| ROCm/HIP | `ROCM_PATH`, `HIP_PATH`, `HIPCXX` (must be Clang, not the `hipcc` wrapper) | + +CMake's `CMAKE_PREFIX_PATH` is also forwarded to the upstream build, so adding the SDK roots there continues to work. + ## Dual native stacks Enabling both local chat (`xlai-sys-llama`, which bundles `ggml` with `llama.cpp`) and native QTS (`xlai-sys-ggml`) links **two** native `ggml` implementations into one binary. Build scripts emit a `cargo:warning` when `xlai-facade` has `llama` + `qts`, or when `xlai-native` enables `qts`. Prefer separate processes or a single stack if you hit duplicate symbols or linker issues. diff --git a/docs/guide/getting-started.md b/docs/guide/getting-started.md index 86b049a..64a85fa 100644 --- a/docs/guide/getting-started.md +++ b/docs/guide/getting-started.md @@ -16,7 +16,7 @@ cd xlai cargo build --workspace ``` -Native crates that wrap `llama.cpp` / QTS now default to `openblas`, `cuda`, `hip`, and `openvino`. On unsupported Apple targets those accelerator flags are ignored with warnings; on supported Linux/Windows hosts you should have the relevant CUDA, ROCm/HIP, and OpenVINO toolchains installed before building. +Native crates that wrap `llama.cpp` / QTS default to requesting `openblas`, `cuda`, `hip`, and `openvino`. The vendored `ggml` / `llama.cpp` core is built and linked **statically**, while accelerator SDK runtimes (CUDA, OpenVINO, ROCm/HIP) are treated as **external system libraries**. The build scripts auto-discover them via `CUDA_PATH` / `OpenVINO_DIR` / `ROCM_PATH` (and standard install paths). When an SDK is missing, the corresponding backend is downgraded with a `cargo:warning` instead of failing the build. `hip` is also currently downgraded on every static-core build because upstream `ggml` does not support `GGML_HIP=ON` together with static linking. See [Native vendor layout](../development/native-vendor) for the full linking contract. ## Tests diff --git a/docs/providers/index.md b/docs/providers/index.md index 0a088af..5787061 100644 --- a/docs/providers/index.md +++ b/docs/providers/index.md @@ -34,4 +34,4 @@ OpenRouter remains chat-only in XLAI today. The dedicated `xlai-backend-openrout ## llama.cpp acceleration matrix -The README tracks the live native accelerator matrix for `llama.cpp` and QTS. Today the workspace defaults the local native stacks to `openblas`, `cuda`, `hip`, and `openvino`, while Vulkan and Metal remain opt-in. See [Support of LLM API providers](https://github.com/yetanother.ai/xlai/blob/main/README.md#support-of-llm-api-providers) on GitHub for the checklist. +The README tracks the live native accelerator matrix for `llama.cpp` and QTS. Today the workspace defaults the local native stacks to request `openblas`, `cuda`, `hip`, and `openvino`, while Vulkan and Metal remain opt-in. The vendored `ggml` / `llama.cpp` core is built statically; CUDA / OpenVINO / ROCm runtimes are linked as **external system SDKs** discovered through `CUDA_PATH` / `OpenVINO_DIR` / `ROCM_PATH`. `hip` is still downgraded to a build warning on every static-core build because upstream `ggml` does not support `GGML_HIP=ON` together with static linking. See [Native vendor layout](../development/native-vendor) for the linking contract and [Support of LLM API providers](https://github.com/yetanother.ai/xlai/blob/main/README.md#support-of-llm-api-providers) on GitHub for the checklist. diff --git a/docs/rust/index.md b/docs/rust/index.md index aaebdab..47100e4 100644 --- a/docs/rust/index.md +++ b/docs/rust/index.md @@ -27,4 +27,4 @@ Concrete examples (OpenAI, llama.cpp, streaming agents) are in the repository [R ## Optional: QTS in native builds -`xlai-native` defaults the local `llama.cpp` stack to `openblas`, `cuda`, `hip`, and `openvino`. Enable the **`qts`** feature when you need `QtsTtsModel`; QTS follows the same default accelerator set, while unsupported Apple-only combinations are skipped with warnings at build time. +`xlai-native` defaults the local `llama.cpp` stack to request `openblas`, `cuda`, `hip`, and `openvino`. Enable the **`qts`** feature when you need `QtsTtsModel`; QTS follows the same requested accelerator set. The vendored `ggml` / `llama.cpp` core is statically linked, while the accelerator SDK runtimes (CUDA / OpenVINO / ROCm) are linked as external system libraries — the build scripts auto-discover them via standard env vars (`CUDA_PATH`, `OpenVINO_DIR`, `ROCM_PATH`). Backends are downgraded with a `cargo:warning` when their preconditions fail (Apple target, missing SDK, or — for `hip` — the upstream constraint that `GGML_HIP` cannot be embedded in a static-core build). See [Native vendor layout](../development/native-vendor) for the full contract. From e81970ed6a55073c0a203f116cd3a4f5be91c510 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Mon, 20 Apr 2026 23:18:48 +0800 Subject: [PATCH 09/12] fixes --- .github/actions/setup-xlai-rust-native/action.yml | 13 ++++++------- crates/sys/xlai-build-native/src/accelerator.rs | 3 ++- docs/development/native-vendor.md | 8 ++++---- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index aed5118..9ea2092 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -110,18 +110,17 @@ runs: - name: Install Windows HIP SDK if: runner.os == 'Windows' shell: pwsh - # Check https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html for the latest installer name + # Check https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html for the latest installer + # name. We use the same `download.amd.com/developer/eula/rocm-hub/...` URL pattern that + # ggml-org/llama.cpp's CI uses (see their `.github/actions/windows-setup-rocm/action.yml`). run: | - $installerName = 'AMD-Software-PRO-Edition-25.Q3-WinSvr2022-For-HIP.exe' - $installerUrl = "https://drivers.amd.com/drivers/prographics/$installerName" + $installerName = 'AMD-Software-PRO-Edition-26.Q1-Win11-For-HIP.exe' + $installerUrl = "https://download.amd.com/developer/eula/rocm-hub/$installerName" $installerPath = Join-Path $env:RUNNER_TEMP $installerName $installLog = Join-Path $env:RUNNER_TEMP 'hip-sdk-install.log' Write-Host "Attempting HIP SDK install from $installerUrl" - Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath -Headers @{ - 'User-Agent' = 'Mozilla/5.0' - 'Referer' = 'https://www.amd.com/' - } + Invoke-WebRequest -Uri $installerUrl -OutFile $installerPath $process = Start-Process -FilePath $installerPath ` -ArgumentList '-install', '-log', $installLog ` diff --git a/crates/sys/xlai-build-native/src/accelerator.rs b/crates/sys/xlai-build-native/src/accelerator.rs index aa037b1..6ac21ff 100644 --- a/crates/sys/xlai-build-native/src/accelerator.rs +++ b/crates/sys/xlai-build-native/src/accelerator.rs @@ -77,7 +77,8 @@ const ROCM_ENV_VARS: &[&str] = &["ROCM_PATH", "HIP_PATH", "ROCM_HOME", "HIPCXX"] /// /// Search order: /// -/// 1. The first non-empty value among [`CUDA_ENV_VARS`]. +/// 1. The first non-empty value among `CUDA_PATH`, `CUDA_HOME`, `CUDA_TOOLKIT_ROOT_DIR`, +/// `CUDAToolkit_ROOT`. /// 2. Standard OS install paths (`/usr/local/cuda*` on Linux, `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v*` on Windows). #[must_use] pub fn detect_cuda_sdk(target: &str) -> Option { diff --git a/docs/development/native-vendor.md b/docs/development/native-vendor.md index da07ac7..75b4805 100644 --- a/docs/development/native-vendor.md +++ b/docs/development/native-vendor.md @@ -52,10 +52,10 @@ Behavior is platform-dependent: The shared helpers honor these environment variables (set them when the SDK lives in a non-default location): -| Backend | Variables | -| -------- | ------------------------------------------------------ | -| CUDA | `CUDA_PATH`, `CUDA_HOME`, `CUDA_TOOLKIT_ROOT_DIR` | -| OpenVINO | `OpenVINO_DIR`, `OPENVINO_ROOT`, `INTEL_OPENVINO_DIR` | +| Backend | Variables | +| -------- | -------------------------------------------------------------------------- | +| CUDA | `CUDA_PATH`, `CUDA_HOME`, `CUDA_TOOLKIT_ROOT_DIR` | +| OpenVINO | `OpenVINO_DIR`, `OPENVINO_ROOT`, `INTEL_OPENVINO_DIR` | | ROCm/HIP | `ROCM_PATH`, `HIP_PATH`, `HIPCXX` (must be Clang, not the `hipcc` wrapper) | CMake's `CMAKE_PREFIX_PATH` is also forwarded to the upstream build, so adding the SDK roots there continues to work. From 80537f8b2f9b17fee80b13cb278882f72218bb95 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Tue, 21 Apr 2026 07:30:31 +0800 Subject: [PATCH 10/12] fixes --- .../actions/setup-xlai-rust-native/action.yml | 110 ++++++++++++++++++ .../sys/xlai-build-native/src/accelerator.rs | 45 ++++--- docs/development/ci-and-testing.md | 4 +- docs/development/crates-taxonomy.md | 2 +- docs/development/native-vendor.md | 6 +- 5 files changed, 146 insertions(+), 21 deletions(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index 9ea2092..0cd2757 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -158,6 +158,116 @@ runs: } Add-Content -Path $env:GITHUB_PATH -Value "$($rocmRoot.FullName)\bin" + - name: Install Linux OpenVINO runtime + if: runner.os == 'Linux' + shell: bash + # Pin to OpenVINO 2026.1.0 archive distribution. Refresh the version + build hash from + # https://github.com/openvinotoolkit/openvino/releases when bumping. Archive layout + # follows storage.openvinotoolkit.org/repositories/openvino/packages//linux/. + run: | + set -euxo pipefail + + source /etc/os-release + case "${VERSION_CODENAME}" in + jammy) variant=ubuntu22 ;; + noble) variant=ubuntu24 ;; + *) + echo "Unsupported Ubuntu codename for OpenVINO archive: ${VERSION_CODENAME}" >&2 + exit 1 + ;; + esac + + OPENVINO_RELEASE=2026.1 + OPENVINO_VERSION=2026.1.0 + OPENVINO_BUILD=2026.1.0.21367.63e31528c62 + archive="openvino_toolkit_${variant}_${OPENVINO_BUILD}_x86_64.tgz" + url="https://storage.openvinotoolkit.org/repositories/openvino/packages/${OPENVINO_RELEASE}/linux/${archive}" + + download_dir="${RUNNER_TEMP}/openvino" + mkdir -p "$download_dir" + curl -fL "$url" --output "${download_dir}/openvino.tgz" + + sudo mkdir -p /opt/intel + sudo tar -xf "${download_dir}/openvino.tgz" -C /opt/intel + sudo mv "/opt/intel/openvino_toolkit_${variant}_${OPENVINO_BUILD}_x86_64" \ + "/opt/intel/openvino_${OPENVINO_VERSION}" + sudo ln -sfn "openvino_${OPENVINO_VERSION}" /opt/intel/openvino_2026 + + sudo -E /opt/intel/openvino_2026/install_dependencies/install_openvino_dependencies.sh -y + + echo "INTEL_OPENVINO_DIR=/opt/intel/openvino_2026" >> "$GITHUB_ENV" + echo "OpenVINO_DIR=/opt/intel/openvino_2026/runtime/cmake" >> "$GITHUB_ENV" + if [ -n "${CMAKE_PREFIX_PATH:-}" ]; then + echo "CMAKE_PREFIX_PATH=/opt/intel/openvino_2026/runtime/cmake:${CMAKE_PREFIX_PATH}" >> "$GITHUB_ENV" + else + echo "CMAKE_PREFIX_PATH=/opt/intel/openvino_2026/runtime/cmake" >> "$GITHUB_ENV" + fi + if [ -n "${LD_LIBRARY_PATH:-}" ]; then + echo "LD_LIBRARY_PATH=/opt/intel/openvino_2026/runtime/lib/intel64:${LD_LIBRARY_PATH}" >> "$GITHUB_ENV" + else + echo "LD_LIBRARY_PATH=/opt/intel/openvino_2026/runtime/lib/intel64" >> "$GITHUB_ENV" + fi + + - name: Install Windows OpenVINO runtime + if: runner.os == 'Windows' + shell: pwsh + # Pin to OpenVINO 2026.1.0 archive distribution. Refresh the version + build hash from + # https://github.com/openvinotoolkit/openvino/releases when bumping. Archive layout + # follows storage.openvinotoolkit.org/repositories/openvino/packages//windows/. + run: | + $openvinoRelease = '2026.1' + $openvinoVersion = '2026.1.0' + $openvinoBuild = '2026.1.0.21367.63e31528c62' + $archiveName = "openvino_toolkit_windows_${openvinoBuild}_x86_64.zip" + $archiveUrl = "https://storage.openvinotoolkit.org/repositories/openvino/packages/${openvinoRelease}/windows/${archiveName}" + $archivePath = Join-Path $env:RUNNER_TEMP $archiveName + $extractRoot = Join-Path $env:RUNNER_TEMP 'openvino-extract' + $intelRoot = 'C:\Program Files (x86)\Intel' + $installRoot = Join-Path $intelRoot "openvino_${openvinoVersion}" + $linkRoot = Join-Path $intelRoot 'openvino_2026' + + Write-Host "Downloading OpenVINO from $archiveUrl" + Invoke-WebRequest -Uri $archiveUrl -OutFile $archivePath + + if (Test-Path $extractRoot) { Remove-Item -Recurse -Force $extractRoot } + New-Item -ItemType Directory -Path $extractRoot | Out-Null + Expand-Archive -Path $archivePath -DestinationPath $extractRoot -Force + + $extractedDir = Get-ChildItem -Path $extractRoot -Directory | Select-Object -First 1 + if (-not $extractedDir) { + throw "OpenVINO archive did not contain a top-level directory under $extractRoot" + } + + if (-not (Test-Path $intelRoot)) { + New-Item -ItemType Directory -Path $intelRoot | Out-Null + } + if (Test-Path $installRoot) { + Remove-Item -Recurse -Force $installRoot + } + Move-Item -Path $extractedDir.FullName -Destination $installRoot + + if (Test-Path $linkRoot) { + Remove-Item -Recurse -Force $linkRoot + } + # Use a junction so we don't need Developer Mode / SeCreateSymbolicLink privileges. + New-Item -ItemType Junction -Path $linkRoot -Target $installRoot | Out-Null + + $cmakeDir = Join-Path $linkRoot 'runtime\cmake' + $libDir = Join-Path $linkRoot 'runtime\bin\intel64\Release' + $tbbBinDir = Join-Path $linkRoot 'runtime\3rdparty\tbb\bin' + + Add-Content -Path $env:GITHUB_ENV -Value "INTEL_OPENVINO_DIR=$linkRoot" + Add-Content -Path $env:GITHUB_ENV -Value "OpenVINO_DIR=$cmakeDir" + if ($env:CMAKE_PREFIX_PATH) { + Add-Content -Path $env:GITHUB_ENV -Value "CMAKE_PREFIX_PATH=$cmakeDir;$env:CMAKE_PREFIX_PATH" + } else { + Add-Content -Path $env:GITHUB_ENV -Value "CMAKE_PREFIX_PATH=$cmakeDir" + } + Add-Content -Path $env:GITHUB_PATH -Value $libDir + if (Test-Path $tbbBinDir) { + Add-Content -Path $env:GITHUB_PATH -Value $tbbBinDir + } + - name: Install Linux build dependencies if: runner.os == 'Linux' shell: bash diff --git a/crates/sys/xlai-build-native/src/accelerator.rs b/crates/sys/xlai-build-native/src/accelerator.rs index 6ac21ff..b4298a6 100644 --- a/crates/sys/xlai-build-native/src/accelerator.rs +++ b/crates/sys/xlai-build-native/src/accelerator.rs @@ -211,29 +211,35 @@ where if value.is_empty() { continue; } - let root = PathBuf::from(value); - let resolved_root = if name == &"HIPCXX" { + let initial = PathBuf::from(value); + let initial = if name == &"HIPCXX" { // HIPCXX usually points at `/llvm/bin/clang`; walk up to the rocm install root. - root.parent() + initial + .parent() .and_then(|p| p.parent()) .and_then(|p| p.parent()) .map(PathBuf::from) - .unwrap_or(root) + .unwrap_or(initial) } else { - root + initial }; - if !resolved_root.is_dir() { - continue; - } - let dirs = lib_dirs(&resolved_root); - if dirs.is_empty() { - continue; + // Try the env value first, then walk up a few parents in case the user pointed at a + // subdir like `/runtime/cmake` (typical for `OpenVINO_DIR`) or `/lib/cmake`. + let mut candidate = Some(initial.clone()); + for _ in 0..4 { + let Some(current) = candidate else { break }; + if current.is_dir() { + let dirs = lib_dirs(¤t); + if !dirs.is_empty() { + return Some(SdkLayout { + root: current, + source: (*name).to_string(), + lib_dirs: dirs, + }); + } + } + candidate = current.parent().map(PathBuf::from); } - return Some(SdkLayout { - root: resolved_root, - source: (*name).to_string(), - lib_dirs: dirs, - }); } None } @@ -247,6 +253,7 @@ fn cuda_lib_dirs(root: &Path, target: &str) -> Vec { push_existing(&mut dirs, root.join("lib64")); push_existing(&mut dirs, root.join("lib")); push_existing(&mut dirs, root.join("targets/x86_64-linux/lib")); + push_existing(&mut dirs, root.join("targets/x86_64-linux/lib/stubs")); push_existing(&mut dirs, root.join("lib/stubs")); push_existing(&mut dirs, root.join("lib64/stubs")); } @@ -364,7 +371,11 @@ fn default_rocm_roots(target: &str) -> Vec<(&'static str, PathBuf)> { } fn cuda_runtime_libs(_target: &str) -> &'static [&'static str] { - &["cudart", "cublas", "cublasLt"] + // `cuda` is the NVIDIA driver library (libcuda.so / cuda.lib). ggml-cuda's VMM pool uses + // the driver API (cuMemCreate / cuMemMap / ...), not just the runtime API in `cudart`. + // The driver lib ships at runtime with the NVIDIA driver; the CUDA toolkit provides a + // build-time stub under `lib64/stubs/` (Linux) or alongside `lib/x64/cuda.lib` (Windows). + &["cudart", "cublas", "cublasLt", "cuda"] } fn openvino_runtime_libs(_target: &str) -> &'static [&'static str] { diff --git a/docs/development/ci-and-testing.md b/docs/development/ci-and-testing.md index 7473c81..9b20416 100644 --- a/docs/development/ci-and-testing.md +++ b/docs/development/ci-and-testing.md @@ -26,7 +26,7 @@ OpenAI smoke tests load `.env` automatically for local runs. For embeddings/tran Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unknown-unknown`, and the `@yai-xlai/xlai` package through the pnpm workspace. -Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. +Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, downloads and installs the OpenVINO runtime archive on Linux and Windows (pinned to a known release, exporting `INTEL_OPENVINO_DIR` / `OpenVINO_DIR` / `CMAKE_PREFIX_PATH`), and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. ### Test (`.github/workflows/test.yml`) @@ -35,7 +35,7 @@ Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs - `cargo test --workspace` - `pnpm --filter @yai-xlai/xlai test` -The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit before building the default native feature set, and the ROCm/HIP setup is applied there as well. The sys-crate `build.rs` files honor `CUDA_PATH` / `OpenVINO_DIR` / `ROCM_PATH` (and standard install layouts) to discover the **external** accelerator SDKs they link against; missing SDKs degrade gracefully with `cargo:warning`. `hip` is currently still downgraded on every static-core build because upstream `ggml` does not allow `GGML_HIP=ON` with static linking, even when ROCm tooling is otherwise available. +The Rust matrix reuses the same shared native setup action, so Linux/Windows unit-test lanes get the CUDA toolkit, the OpenVINO runtime archive, and the ROCm/HIP SDK before building the default native feature set. The sys-crate `build.rs` files honor `CUDA_PATH` / `OpenVINO_DIR` / `INTEL_OPENVINO_DIR` / `ROCM_PATH` (and standard install layouts) to discover the **external** accelerator SDKs they link against; missing SDKs degrade gracefully with `cargo:warning`. `hip` is currently still downgraded on every static-core build because upstream `ggml` does not allow `GGML_HIP=ON` with static linking, even when ROCm tooling is otherwise available. ### Publish (`.github/workflows/publish.yml`) diff --git a/docs/development/crates-taxonomy.md b/docs/development/crates-taxonomy.md index 13c0528..62a9ac7 100644 --- a/docs/development/crates-taxonomy.md +++ b/docs/development/crates-taxonomy.md @@ -42,7 +42,7 @@ This doc classifies crates for dependency and release decisions. Authoritative p ## CI setup -Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, ROCm/HIP tooling on Linux and Windows, plus OpenBLAS / vcpkg / shaderc). The action also prints the resolved accelerator SDK environment so CI logs reflect the static-core plus external-SDK linking contract. Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. `hip` remains downgraded with a warning on the static-core sys-crate path because upstream `ggml` does not support `GGML_HIP=ON` with `BUILD_SHARED_LIBS=OFF` / `GGML_STATIC=ON`. +Native Rust jobs share [`.github/actions/setup-xlai-rust-native`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) (toolchain, sccache, `rust-cache`, CUDA on Linux/Windows via `Jimver/cuda-toolkit`, ROCm/HIP tooling on Linux and Windows, the OpenVINO runtime archive on Linux and Windows, plus OpenBLAS / vcpkg / shaderc). The action also prints the resolved accelerator SDK environment so CI logs reflect the static-core plus external-SDK linking contract. Jobs that need **Vulkan** (release `build.yml`) add Linux `glslc` / `libvulkan-dev` and the Vulkan SDK step after that action. `hip` remains downgraded with a warning on the static-core sys-crate path because upstream `ggml` does not support `GGML_HIP=ON` with `BUILD_SHARED_LIBS=OFF` / `GGML_STATIC=ON`. ## Adding a new workspace member diff --git a/docs/development/native-vendor.md b/docs/development/native-vendor.md index 75b4805..f3b51eb 100644 --- a/docs/development/native-vendor.md +++ b/docs/development/native-vendor.md @@ -55,11 +55,15 @@ The shared helpers honor these environment variables (set them when the SDK live | Backend | Variables | | -------- | -------------------------------------------------------------------------- | | CUDA | `CUDA_PATH`, `CUDA_HOME`, `CUDA_TOOLKIT_ROOT_DIR` | -| OpenVINO | `OpenVINO_DIR`, `OPENVINO_ROOT`, `INTEL_OPENVINO_DIR` | +| OpenVINO | `OpenVINO_DIR`, `OPENVINO_ROOT`, `INTEL_OPENVINO_DIR`, `OPENVINO_HOME` | | ROCm/HIP | `ROCM_PATH`, `HIP_PATH`, `HIPCXX` (must be Clang, not the `hipcc` wrapper) | +For the OpenVINO variables, the helpers tolerate values that point at a sub-directory such as `/runtime/cmake` (the CMake convention for `OpenVINO_DIR`) by walking up parent directories until a `runtime/lib/...` layout is found. Setting any one of `INTEL_OPENVINO_DIR`, `OPENVINO_ROOT`, `OPENVINO_HOME` to the install root, or `OpenVINO_DIR` to the cmake config directory, is sufficient. + CMake's `CMAKE_PREFIX_PATH` is also forwarded to the upstream build, so adding the SDK roots there continues to work. +CI installs OpenVINO automatically on Linux and Windows from `storage.openvinotoolkit.org` (pinned release; see [`.github/actions/setup-xlai-rust-native/action.yml`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) for the current version), so external SDK linking is exercised on every native build lane. + ## Dual native stacks Enabling both local chat (`xlai-sys-llama`, which bundles `ggml` with `llama.cpp`) and native QTS (`xlai-sys-ggml`) links **two** native `ggml` implementations into one binary. Build scripts emit a `cargo:warning` when `xlai-facade` has `llama` + `qts`, or when `xlai-native` enables `qts`. Prefer separate processes or a single stack if you hit duplicate symbols or linker issues. From 167235b0756140c8998bcae0020cdacef63cf02d Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Wed, 22 Apr 2026 12:55:43 +0800 Subject: [PATCH 11/12] ag --- .../actions/setup-xlai-rust-native/action.yml | 45 +++++++++++++++++++ docs/development/ci-and-testing.md | 2 +- docs/development/native-vendor.md | 2 + 3 files changed, 48 insertions(+), 1 deletion(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index 0cd2757..f39096d 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -255,6 +255,51 @@ runs: $cmakeDir = Join-Path $linkRoot 'runtime\cmake' $libDir = Join-Path $linkRoot 'runtime\bin\intel64\Release' $tbbBinDir = Join-Path $linkRoot 'runtime\3rdparty\tbb\bin' + $runtimeIncludeDir = Join-Path $linkRoot 'runtime\include' + + # The OpenVINO Windows archive does not bundle OpenCL headers, but + # `` transitively includes + # `intel_gpu/ocl/ocl_wrapper.hpp`, which requires `CL/cl2.hpp`, + # `CL/cl.h`, etc. On Linux the equivalent headers come from the + # `ocl-icd-opencl-dev` (or similar) package installed by + # `install_openvino_dependencies.sh`. Pull the official Khronos + # OpenCL-Headers + OpenCL-CLHPP releases and drop the `CL/` tree + # into OpenVINO's `runtime/include/` so the existing + # `-I/runtime/include` flag finds them transparently. + $openclHeadersUrl = 'https://github.com/KhronosGroup/OpenCL-Headers/archive/refs/tags/v2025.07.22.zip' + $openclClhppUrl = 'https://github.com/KhronosGroup/OpenCL-CLHPP/archive/refs/tags/v2025.07.22.zip' + $openclWorkDir = Join-Path $env:RUNNER_TEMP 'opencl-headers' + if (Test-Path $openclWorkDir) { Remove-Item -Recurse -Force $openclWorkDir } + New-Item -ItemType Directory -Path $openclWorkDir | Out-Null + + $openclHeadersZip = Join-Path $openclWorkDir 'opencl-headers.zip' + Write-Host "Downloading OpenCL C headers from $openclHeadersUrl" + Invoke-WebRequest -Uri $openclHeadersUrl -OutFile $openclHeadersZip + Expand-Archive -Path $openclHeadersZip -DestinationPath $openclWorkDir -Force + $openclHeadersDir = Get-ChildItem -Path $openclWorkDir -Directory ` + | Where-Object { $_.Name -like 'OpenCL-Headers*' } ` + | Select-Object -First 1 + if (-not $openclHeadersDir) { throw 'OpenCL-Headers archive layout unexpected' } + + $openclClhppZip = Join-Path $openclWorkDir 'opencl-clhpp.zip' + Write-Host "Downloading OpenCL C++ headers from $openclClhppUrl" + Invoke-WebRequest -Uri $openclClhppUrl -OutFile $openclClhppZip + Expand-Archive -Path $openclClhppZip -DestinationPath $openclWorkDir -Force + $openclClhppDir = Get-ChildItem -Path $openclWorkDir -Directory ` + | Where-Object { $_.Name -like 'OpenCL-CLHPP*' } ` + | Select-Object -First 1 + if (-not $openclClhppDir) { throw 'OpenCL-CLHPP archive layout unexpected' } + + $openvinoClDir = Join-Path $runtimeIncludeDir 'CL' + if (-not (Test-Path $openvinoClDir)) { + New-Item -ItemType Directory -Path $openvinoClDir | Out-Null + } + Copy-Item -Force -Recurse ` + -Path (Join-Path $openclHeadersDir.FullName 'CL\*') ` + -Destination $openvinoClDir + Copy-Item -Force -Recurse ` + -Path (Join-Path $openclClhppDir.FullName 'include\CL\*') ` + -Destination $openvinoClDir Add-Content -Path $env:GITHUB_ENV -Value "INTEL_OPENVINO_DIR=$linkRoot" Add-Content -Path $env:GITHUB_ENV -Value "OpenVINO_DIR=$cmakeDir" diff --git a/docs/development/ci-and-testing.md b/docs/development/ci-and-testing.md index 9b20416..4c5f4c6 100644 --- a/docs/development/ci-and-testing.md +++ b/docs/development/ci-and-testing.md @@ -26,7 +26,7 @@ OpenAI smoke tests load `.env` automatically for local runs. For embeddings/tran Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unknown-unknown`, and the `@yai-xlai/xlai` package through the pnpm workspace. -Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, downloads and installs the OpenVINO runtime archive on Linux and Windows (pinned to a known release, exporting `INTEL_OPENVINO_DIR` / `OpenVINO_DIR` / `CMAKE_PREFIX_PATH`), and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. +Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, downloads and installs the OpenVINO runtime archive on Linux and Windows (pinned to a known release, exporting `INTEL_OPENVINO_DIR` / `OpenVINO_DIR` / `CMAKE_PREFIX_PATH`), drops the official Khronos OpenCL C/C++ headers into the OpenVINO `runtime/include/CL/` tree on Windows (the OpenVINO archive itself does not bundle them, but `intel_gpu/ocl/ocl_wrapper.hpp` requires `CL/cl2.hpp`), and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. ### Test (`.github/workflows/test.yml`) diff --git a/docs/development/native-vendor.md b/docs/development/native-vendor.md index f3b51eb..23fb68a 100644 --- a/docs/development/native-vendor.md +++ b/docs/development/native-vendor.md @@ -64,6 +64,8 @@ CMake's `CMAKE_PREFIX_PATH` is also forwarded to the upstream build, so adding t CI installs OpenVINO automatically on Linux and Windows from `storage.openvinotoolkit.org` (pinned release; see [`.github/actions/setup-xlai-rust-native/action.yml`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) for the current version), so external SDK linking is exercised on every native build lane. +> **Windows + OpenVINO + OpenCL headers.** The OpenVINO Windows archive does not bundle OpenCL headers, but `` transitively pulls in `intel_gpu/ocl/ocl_wrapper.hpp`, which requires `CL/cl2.hpp`. On Linux this is satisfied by the system OpenCL package installed by `install_openvino_dependencies.sh`. The CI setup action drops the official Khronos `OpenCL-Headers` and `OpenCL-CLHPP` releases into `/runtime/include/CL/` so the existing OpenVINO include path resolves them. Local Windows builds need the same headers reachable via either `INCLUDE`, `CMAKE_INCLUDE_PATH`, or alongside the OpenVINO runtime headers (e.g. via `vcpkg install opencl:x64-windows-static-md` plus a manual copy into `/runtime/include/CL/`). + ## Dual native stacks Enabling both local chat (`xlai-sys-llama`, which bundles `ggml` with `llama.cpp`) and native QTS (`xlai-sys-ggml`) links **two** native `ggml` implementations into one binary. Build scripts emit a `cargo:warning` when `xlai-facade` has `llama` + `qts`, or when `xlai-native` enables `qts`. Prefer separate processes or a single stack if you hit duplicate symbols or linker issues. From 07e3d915dd9d178238edd7cc08c88519252c8ef4 Mon Sep 17 00:00:00 2001 From: Delton Ding Date: Wed, 22 Apr 2026 13:08:16 +0800 Subject: [PATCH 12/12] fixes --- .github/actions/setup-xlai-rust-native/action.yml | 14 +++++++++++++- docs/development/ci-and-testing.md | 2 +- docs/development/native-vendor.md | 5 ++++- 3 files changed, 18 insertions(+), 3 deletions(-) diff --git a/.github/actions/setup-xlai-rust-native/action.yml b/.github/actions/setup-xlai-rust-native/action.yml index f39096d..e6d7160 100644 --- a/.github/actions/setup-xlai-rust-native/action.yml +++ b/.github/actions/setup-xlai-rust-native/action.yml @@ -316,9 +316,21 @@ runs: - name: Install Linux build dependencies if: runner.os == 'Linux' shell: bash + # `ocl-icd-opencl-dev` (+ `opencl-headers`) is required so that ggml-openvino's + # `find_package(OpenCL)` resolves on Linux. The OpenVINO archive's + # `install_openvino_dependencies.sh` only installs runtime libs, not the dev + # package that exposes `OpenCL_LIBRARY` / `OpenCL_INCLUDE_DIR` to CMake. + # `opencl-clhpp-headers` provides the C++ wrappers (`CL/cl2.hpp`, + # `CL/opencl.hpp`) used transitively by `` via + # `intel_gpu/ocl/ocl_wrapper.hpp`. run: | sudo apt-get update - sudo apt-get install -y libopenblas-dev libasound2-dev + sudo apt-get install -y \ + libopenblas-dev \ + libasound2-dev \ + ocl-icd-opencl-dev \ + opencl-headers \ + opencl-clhpp-headers - name: Install Windows OpenBLAS (vcpkg) if: runner.os == 'Windows' diff --git a/docs/development/ci-and-testing.md b/docs/development/ci-and-testing.md index 4c5f4c6..f75a715 100644 --- a/docs/development/ci-and-testing.md +++ b/docs/development/ci-and-testing.md @@ -26,7 +26,7 @@ OpenAI smoke tests load `.env` automatically for local runs. For embeddings/tran Builds on Linux, Windows, macOS arm64, macOS x86_64 (cross-target), `wasm32-unknown-unknown`, and the `@yai-xlai/xlai` package through the pnpm workspace. -Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, downloads and installs the OpenVINO runtime archive on Linux and Windows (pinned to a known release, exporting `INTEL_OPENVINO_DIR` / `OpenVINO_DIR` / `CMAKE_PREFIX_PATH`), drops the official Khronos OpenCL C/C++ headers into the OpenVINO `runtime/include/CL/` tree on Windows (the OpenVINO archive itself does not bundle them, but `intel_gpu/ocl/ocl_wrapper.hpp` requires `CL/cl2.hpp`), and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. +Native Rust jobs share `.github/actions/setup-xlai-rust-native`, which installs the Rust toolchain, caches cargo output, provisions OpenBLAS, installs the CUDA toolkit on Linux/Windows via `Jimver/cuda-toolkit@v0.2`, installs ROCm/HIP tooling on Linux and Windows, downloads and installs the OpenVINO runtime archive on Linux and Windows (pinned to a known release, exporting `INTEL_OPENVINO_DIR` / `OpenVINO_DIR` / `CMAKE_PREFIX_PATH`), installs the OpenCL dev / C++ wrapper headers (`ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers` via apt on Linux; the official Khronos `OpenCL-Headers` and `OpenCL-CLHPP` releases dropped into `/runtime/include/CL/` on Windows) so `ggml-openvino`'s `find_package(OpenCL)` and ``'s `intel_gpu/ocl/ocl_wrapper.hpp` (which needs `CL/cl2.hpp`) both resolve, and finally prints the resolved accelerator SDK env vars so the static-core plus external-SDK linking behavior is visible in build logs. Vulkan remains an extra workflow step because only the Vulkan lanes need `glslc` / the SDK. ### Test (`.github/workflows/test.yml`) diff --git a/docs/development/native-vendor.md b/docs/development/native-vendor.md index 23fb68a..beee718 100644 --- a/docs/development/native-vendor.md +++ b/docs/development/native-vendor.md @@ -64,7 +64,10 @@ CMake's `CMAKE_PREFIX_PATH` is also forwarded to the upstream build, so adding t CI installs OpenVINO automatically on Linux and Windows from `storage.openvinotoolkit.org` (pinned release; see [`.github/actions/setup-xlai-rust-native/action.yml`](https://github.com/yetanother.ai/xlai/blob/main/.github/actions/setup-xlai-rust-native/action.yml) for the current version), so external SDK linking is exercised on every native build lane. -> **Windows + OpenVINO + OpenCL headers.** The OpenVINO Windows archive does not bundle OpenCL headers, but `` transitively pulls in `intel_gpu/ocl/ocl_wrapper.hpp`, which requires `CL/cl2.hpp`. On Linux this is satisfied by the system OpenCL package installed by `install_openvino_dependencies.sh`. The CI setup action drops the official Khronos `OpenCL-Headers` and `OpenCL-CLHPP` releases into `/runtime/include/CL/` so the existing OpenVINO include path resolves them. Local Windows builds need the same headers reachable via either `INCLUDE`, `CMAKE_INCLUDE_PATH`, or alongside the OpenVINO runtime headers (e.g. via `vcpkg install opencl:x64-windows-static-md` plus a manual copy into `/runtime/include/CL/`). +> **OpenVINO + OpenCL headers.** The `ggml-openvino` backend calls `find_package(OpenCL)` and `` transitively pulls in `intel_gpu/ocl/ocl_wrapper.hpp`, which requires `CL/cl2.hpp`. The OpenVINO archives themselves do not ship OpenCL headers / loader, so the build environment must provide both: +> +> - **Linux**: install `ocl-icd-opencl-dev opencl-headers opencl-clhpp-headers` via apt (or the equivalent for your distro). `install_openvino_dependencies.sh` does not cover the dev / C++ wrapper packages. +> - **Windows**: `find_package(OpenCL)` typically resolves via the bundled CUDA toolkit (`OpenCL.lib` + `CL/cl.h`), but the C++ wrappers are missing. The CI setup action drops the official Khronos `OpenCL-Headers` and `OpenCL-CLHPP` releases into `/runtime/include/CL/` so the existing `-I/runtime/include` flag resolves them transparently. Local Windows builds need the same headers reachable on the include search path (e.g. via `vcpkg install opencl-clhpp:x64-windows-static-md` plus a manual copy into `/runtime/include/CL/`). ## Dual native stacks