Refactor: Migrate from FastChat to vLLM with native XPU backend#7
Merged
Merged
Conversation
The current stack (PyTorch 2.0.1a0 + IPEX 2.0.110+xpu + FastChat + llama-cpp-python/CLBlast) depends on packages from mid-2023 and an upstream ecosystem (intel/ipex-llm, intel-extension-for-pytorch) that is now archived or end-of-life. The plan proposes migrating to vLLM with native Intel XPU support on top of intel/intel-extension-for-pytorch: 2.8.10-xpu, which supports both Arc A-series and B-series GPUs, ships an OpenAI-compatible API, and enables PagedAttention, continuous batching, and INT4/FP8 quantization. Reduces the Dockerfile from 105 to ~25 lines, replaces the 4-process FastChat startup with a single vLLM process, and fixes missing GPU env vars (SYCL_CACHE_PERSISTENT, USE_XETLA, ZES_ENABLE_SYSMAN, etc).
Migrates the container from FastChat + IPEX 2.0.110 + llama-cpp-python/CLBlast to vLLM with the native Intel XPU backend, running on top of Intel's intel/intel-extension-for-pytorch:2.8.10-xpu base image. This covers both Arc A-series (A770/A750) and B-series (B580, Pro B60/B70) GPUs, ships an OpenAI-compatible API natively on port 8000, and enables PagedAttention, continuous batching, and INT4/FP8 quantization. Dockerfile drops from ~105 to ~40 lines (base image handles the GPU driver stack and oneAPI). startup.sh drops from 47 to ~15 lines; the 4-process FastChat topology (controller + worker + gradio + openai) collapses to a single vLLM api_server process. Adds missing Intel-documented GPU env vars (SYCL_CACHE_PERSISTENT, USE_XETLA=OFF, SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS, ZES_ENABLE_SYSMAN, UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS). Reintroduces docker-compose.yaml with the required group_add/ipc/shm_size settings that the previous docker-run-only docs omitted. GitHub Actions bumped to v4/v3/v6 with build caching. README notes an upcoming rename to itlackey/vllm-arc in the next release; the ipex-arc-fastchat tag will be published for one more release then stop.
There was a problem hiding this comment.
Pull request overview
This PR modernizes the Intel Arc GPU inference container by replacing the legacy FastChat + IPEX stack with a single-process vLLM OpenAI-compatible server using vLLM’s native XPU backend, along with updated container build/run workflows and documentation.
Changes:
- Replaced FastChat multi-process startup with a single vLLM OpenAI API server entrypoint.
- Rewrote the Docker image to use
intel/intel-extension-for-pytorch:2.8.10-xpuand install vLLM XPU support. - Added a Docker Compose setup and updated CI workflows + README to match the new runtime model.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
startup.sh |
Simplifies startup to exec a single vLLM OpenAI API server process with XPU device. |
Dockerfile |
Switches base image to Intel’s IPEX PyTorch XPU image; installs vLLM XPU and sets runtime env. |
docker-compose.yaml |
Adds a compose service for vLLM with /dev/dri mapping, groups, IPC/SHM settings, and HF cache mount. |
README.md |
Rewrites usage docs for vLLM/OpenAI API, adds compose instructions, tuning guidance, and Open WebUI pairing. |
.github/workflows/build-docker-image.yml |
Updates CI build workflow to Buildx + build-push-action with GHA caching. |
.github/workflows/build-push-image.yml |
Updates release publishing workflow to Buildx + build-push-action with GHA caching. |
REFACTOR_PLAN.md |
Adds a detailed design/refactor plan and testing checklist documenting the migration approach. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| ports: | ||
| - "8000:8000" | ||
| volumes: | ||
| - ${HF_CACHE:-~/.cache/huggingface}:/root/.cache/huggingface |
There was a problem hiding this comment.
The host path default ${HF_CACHE:-~/.cache/huggingface} relies on ~ expansion, which Docker Compose does not reliably expand (it’s treated as a literal ~ on many setups). Prefer an explicit absolute path (e.g., ${HF_CACHE:-${HOME}/.cache/huggingface}) to avoid mounting an unintended directory.
Suggested change
| - ${HF_CACHE:-~/.cache/huggingface}:/root/.cache/huggingface | |
| - ${HF_CACHE:-${HOME}/.cache/huggingface}:/root/.cache/huggingface |
Dockerfile: - Switch base to intel/vllm:0.14.1-xpu (prebuilt; fixes CVE-2026-22778 CVSS 9.8 RCE present in 0.14.0; avoids vllm[xpu] pip extra which does not exist on PyPI and would silently install the CPU build over an incompatible PyTorch version) - Add UR_L0_USE_IMMEDIATE_COMMANDLISTS=1 (L0 V2 adapter equivalent for B-series / oneAPI 2025.3+; previous var only covered legacy adapter / A-series) - Add VLLM_WORKER_MULTIPROC_METHOD=spawn (SYCL contexts are not fork-safe; prevents deadlocks on multi-GPU tensor-parallel setups) - Add HEALTHCHECK on /health endpoint with 120s start_period for model load time - Add --max-model-len 8192 to default CMD (matches compose override) - Remove pip install step (vllm pre-installed in base image) docker-compose.yaml: - Remove ipc: host (nullified shm_size; shm_size alone is sufficient for single-GPU; ipc: host still documented for multi-GPU in README) - Add healthcheck block (retries=20, start_period=120s for model load latency) - Switch group_add to RENDER_GID/VIDEO_GID env vars to fix host/container GID mismatch that blocks /dev/dri/renderD128 access on many distros CI workflows: - Bump action versions: checkout@v4->v6, setup-buildx@v3->v4, login@v3->v4, build-push@v6->v7 - Fix workflow_dispatch tag_name bug: empty github.event.release.tag_name on manual dispatch produced an invalid Docker tag; fallback to github.sha - Add permissions blocks (contents: read; packages: write; id-token: write) - Add provenance: mode=max and sbom: true to publish workflow (SLSA L2) - Extend build trigger to all branches (not just main) for earlier validation README.md: - Fix kernel version: B-series requires 6.12+ (not 6.2+) - Add host driver installation section with package names and usermod command - Add A-series compatibility warning re: vLLM >= 0.10.0 attention backend - Fix GPU memory table: add Arc A770 8GB SKU row; A770 has two VRAM variants - Fix bfloat16 note: silently falls back to float16 on A-series (Alchemist) - Fix quantization: awq_marlin/gptq_marlin use CUDA-only Marlin kernels and are not supported on Intel XPU; document kv-cache-dtype fp8 as the verified XPU alternative - Fix docker run example: add --shm-size, remove --ipc=host, add HF_TOKEN, add --max-model-len; update shm explanation - Fix Open WebUI snippet: :main -> :latest - Update dev build arg: VLLM_VERSION -> VLLM_TAG - Add Security section: unauthenticated 0.0.0.0, --api-key guidance, /metrics exposure, HF_TOKEN secret management warning .gitignore: add .env (prevent accidental HF_TOKEN commit) .env.example: new file documenting HF_TOKEN, HF_CACHE, RENDER_GID, VIDEO_GID REFACTOR_PLAN.md: - Status: Proposed -> Implemented - Update base image section to reflect intel/vllm prebuilt approach - Expand risk table with 7 missing entries: CVE-2026-22778, auth exposure, root+ipc privilege escalation, awq_marlin CUDA-only, B-series kernel req, HF_TOKEN exposure, IPEX EOL timeline - Fix quantization row in decision matrix - Fix testing checklist (awq_marlin -> kv-cache-dtype fp8)
Qwen3-4B is the optimal Qwen3 model for A770 16 GB: - ~8 GB float16 weights vs ~14 GB for Qwen2.5-7B-Instruct - ~6.4 GB free for KV cache at max-model-len 8192 (vs nearly none for 7B) - Quality matches Qwen2.5-7B-Instruct in standard mode; significantly better with chain-of-thought (thinking mode) - Qwen3ForCausalLM architecture confirmed supported in vLLM 0.14.1 XPU - Qwen3-8B (~16.4 GB float16) cannot physically fit on A770 16 GB at fp16 Also updates multi-GPU example to Qwen3-32B.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR replaces the FastChat + IPEX 2.0.110 + CLBlast stack with vLLM's native Intel XPU backend, modernizing the inference server while adding support for Arc B-series GPUs and improving performance through PagedAttention and continuous batching.
Key Changes
Dockerfile rewrite: Reduced from 105 to ~25 lines by using
intel/intel-extension-for-pytorch:2.8.10-xpuas the base image (which includes PyTorch 2.8, XPU support, and the complete Intel GPU driver stack). Removed manual oneAPI repository setup, IPEX 2.0.110 pins, llama-cpp-python with CLBlast, and FastChat dependencies.startup.sh simplification: Reduced from 47 to ~12 lines. Replaced the multi-process FastChat architecture (controller + worker + gradio + openai_api_server) with a single vLLM process. Removed awk-based model-name parsing and health-check polling loops; vLLM handles all CLI argument parsing natively.
Added docker-compose.yaml: New compose file with proper GPU device mapping, group assignments (
video,render), IPC host mode, and shared memory configuration required by vLLM's PagedAttention. Includes environment variable support for HuggingFace token and model caching.README rewrite: Updated documentation to reflect vLLM's OpenAI-compatible API, removed FastChat and Gradio UI references, added GPU memory sizing table, documented multi-GPU tensor parallelism, and included Open WebUI as the recommended web UI pairing.
CI workflow updates: Modernized GitHub Actions workflows to use
docker/setup-buildx-action@v3anddocker/build-push-action@v6with proper BuildKit caching.Notable Implementation Details
GPU support expansion: Now covers both Arc A-series (A770, A750) and B-series (B580, Arc Pro B60/B70) through vLLM's XPU backend, whereas the previous IPEX 2.0.110 predates B-series support.
Quantization support: vLLM enables INT4/INT8/FP8/AWQ/GPTQ quantization via CLI flags (
--quantization awq_marlin), whereas the previous stack had no quantization support.Performance improvements: PagedAttention and continuous batching provide significantly higher throughput on concurrent requests compared to FastChat's sequential processing.
Environment variables: Added Intel-documented SYCL runtime tuning (
SYCL_CACHE_PERSISTENT,SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS,UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS) to optimize Arc GPU performance and avoid per-startup kernel recompilation.Backward compatibility: Docker image tag remains
itlackey/ipex-arc-fastchat:latestfor one release to avoid breaking existing deployments; a rename tovllm-arcis planned for the next major version.Testing Recommendations
https://claude.ai/code/session_01TRPRHbSoEeXkwAgkAr8GA9