Release/0.0.6#9
Open
itlackey wants to merge 8 commits into
Open
Conversation
The current stack (PyTorch 2.0.1a0 + IPEX 2.0.110+xpu + FastChat + llama-cpp-python/CLBlast) depends on packages from mid-2023 and an upstream ecosystem (intel/ipex-llm, intel-extension-for-pytorch) that is now archived or end-of-life. The plan proposes migrating to vLLM with native Intel XPU support on top of intel/intel-extension-for-pytorch: 2.8.10-xpu, which supports both Arc A-series and B-series GPUs, ships an OpenAI-compatible API, and enables PagedAttention, continuous batching, and INT4/FP8 quantization. Reduces the Dockerfile from 105 to ~25 lines, replaces the 4-process FastChat startup with a single vLLM process, and fixes missing GPU env vars (SYCL_CACHE_PERSISTENT, USE_XETLA, ZES_ENABLE_SYSMAN, etc).
Migrates the container from FastChat + IPEX 2.0.110 + llama-cpp-python/CLBlast to vLLM with the native Intel XPU backend, running on top of Intel's intel/intel-extension-for-pytorch:2.8.10-xpu base image. This covers both Arc A-series (A770/A750) and B-series (B580, Pro B60/B70) GPUs, ships an OpenAI-compatible API natively on port 8000, and enables PagedAttention, continuous batching, and INT4/FP8 quantization. Dockerfile drops from ~105 to ~40 lines (base image handles the GPU driver stack and oneAPI). startup.sh drops from 47 to ~15 lines; the 4-process FastChat topology (controller + worker + gradio + openai) collapses to a single vLLM api_server process. Adds missing Intel-documented GPU env vars (SYCL_CACHE_PERSISTENT, USE_XETLA=OFF, SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS, ZES_ENABLE_SYSMAN, UR_L0_ENABLE_RELAXED_ALLOCATION_LIMITS). Reintroduces docker-compose.yaml with the required group_add/ipc/shm_size settings that the previous docker-run-only docs omitted. GitHub Actions bumped to v4/v3/v6 with build caching. README notes an upcoming rename to itlackey/vllm-arc in the next release; the ipex-arc-fastchat tag will be published for one more release then stop.
Dockerfile: - Switch base to intel/vllm:0.14.1-xpu (prebuilt; fixes CVE-2026-22778 CVSS 9.8 RCE present in 0.14.0; avoids vllm[xpu] pip extra which does not exist on PyPI and would silently install the CPU build over an incompatible PyTorch version) - Add UR_L0_USE_IMMEDIATE_COMMANDLISTS=1 (L0 V2 adapter equivalent for B-series / oneAPI 2025.3+; previous var only covered legacy adapter / A-series) - Add VLLM_WORKER_MULTIPROC_METHOD=spawn (SYCL contexts are not fork-safe; prevents deadlocks on multi-GPU tensor-parallel setups) - Add HEALTHCHECK on /health endpoint with 120s start_period for model load time - Add --max-model-len 8192 to default CMD (matches compose override) - Remove pip install step (vllm pre-installed in base image) docker-compose.yaml: - Remove ipc: host (nullified shm_size; shm_size alone is sufficient for single-GPU; ipc: host still documented for multi-GPU in README) - Add healthcheck block (retries=20, start_period=120s for model load latency) - Switch group_add to RENDER_GID/VIDEO_GID env vars to fix host/container GID mismatch that blocks /dev/dri/renderD128 access on many distros CI workflows: - Bump action versions: checkout@v4->v6, setup-buildx@v3->v4, login@v3->v4, build-push@v6->v7 - Fix workflow_dispatch tag_name bug: empty github.event.release.tag_name on manual dispatch produced an invalid Docker tag; fallback to github.sha - Add permissions blocks (contents: read; packages: write; id-token: write) - Add provenance: mode=max and sbom: true to publish workflow (SLSA L2) - Extend build trigger to all branches (not just main) for earlier validation README.md: - Fix kernel version: B-series requires 6.12+ (not 6.2+) - Add host driver installation section with package names and usermod command - Add A-series compatibility warning re: vLLM >= 0.10.0 attention backend - Fix GPU memory table: add Arc A770 8GB SKU row; A770 has two VRAM variants - Fix bfloat16 note: silently falls back to float16 on A-series (Alchemist) - Fix quantization: awq_marlin/gptq_marlin use CUDA-only Marlin kernels and are not supported on Intel XPU; document kv-cache-dtype fp8 as the verified XPU alternative - Fix docker run example: add --shm-size, remove --ipc=host, add HF_TOKEN, add --max-model-len; update shm explanation - Fix Open WebUI snippet: :main -> :latest - Update dev build arg: VLLM_VERSION -> VLLM_TAG - Add Security section: unauthenticated 0.0.0.0, --api-key guidance, /metrics exposure, HF_TOKEN secret management warning .gitignore: add .env (prevent accidental HF_TOKEN commit) .env.example: new file documenting HF_TOKEN, HF_CACHE, RENDER_GID, VIDEO_GID REFACTOR_PLAN.md: - Status: Proposed -> Implemented - Update base image section to reflect intel/vllm prebuilt approach - Expand risk table with 7 missing entries: CVE-2026-22778, auth exposure, root+ipc privilege escalation, awq_marlin CUDA-only, B-series kernel req, HF_TOKEN exposure, IPEX EOL timeline - Fix quantization row in decision matrix - Fix testing checklist (awq_marlin -> kv-cache-dtype fp8)
Qwen3-4B is the optimal Qwen3 model for A770 16 GB: - ~8 GB float16 weights vs ~14 GB for Qwen2.5-7B-Instruct - ~6.4 GB free for KV cache at max-model-len 8192 (vs nearly none for 7B) - Quality matches Qwen2.5-7B-Instruct in standard mode; significantly better with chain-of-thought (thinking mode) - Qwen3ForCausalLM architecture confirmed supported in vLLM 0.14.1 XPU - Qwen3-8B (~16.4 GB float16) cannot physically fit on A770 16 GB at fp16 Also updates multi-GPU example to Qwen3-32B.
Refactor: Migrate from FastChat to vLLM with native XPU backend
Upgrades the default model from Qwen3-4B to Qwen3.5-4B across docker-compose.yaml, Dockerfile, and README examples. https://claude.ai/code/session_01K4HGfjFPSf3ViGXveLYPY9
Adds three Docker Compose override files: - docker-compose.16gb.yaml: Qwen3.5-9B at float16, 4K context (needs INT4/INT8 quant) - docker-compose.24gb.yaml: Qwen3.5-9B at bfloat16, 8K context (fits without quant) - docker-compose.32gb.yaml: Qwen3.6-35B-A3B at bfloat16, tensor-parallel-size 2 (needs INT4 quant) Updates README GPU memory table and adds a GPU profiles section documenting quantization requirements and the compose -f merge pattern for each tier. https://claude.ai/code/session_01K4HGfjFPSf3ViGXveLYPY9
Claude/integrate qwen model af x zs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request makes a major transition from a FastChat + IPEX-LLM stack to a vLLM-based OpenAI-compatible inference server for Intel Arc GPUs, with significant improvements to GPU support, quantization options, and maintainability. The Dockerfile is rebuilt on Intel’s official vLLM XPU image, the documentation is fully rewritten to reflect the new stack and usage, and the CI/CD pipeline is modernized for better reliability and caching. The update also introduces support for both A-series and B-series Arc GPUs, and provides new configuration and security recommendations.
Core platform migration and GPU support:
vllm:0.14.1-xpuDocker image, enabling native support for both A-series and B-series Arc GPUs, with improved performance, quantization options (INT4/INT8/FP8/AWQ/GPTQ), and simplified container startup. [1] [2]Documentation overhaul:
README.mdto document the new vLLM-based stack, including quickstart instructions, GPU compatibility notes, quantization guidance, Docker Compose profiles for different GPU tiers, API usage examples, and security best practices..env.examplewith recommended environment variables for HuggingFace tokens, model cache, and required group IDs for GPU access.Container and build improvements:
Dockerfileby removing custom dependency installation in favor of the Intel-maintained base image, added healthchecks, and streamlined the startup script.CI/CD modernization:
Security and operational guidance:
These changes collectively modernize the project, improve GPU support and performance, and make deployment and usage more robust and secure.