Serving Diffusion Models

This repo shows how to serve image-editing diffusion models (e.g. Qwen/Qwen-Image-Edit-2511) through two interchangeable, production-style backends:

Method	Engine	Auth	Folder
vLLM Omni	vllm/vllm-omni (OpenAI-compatible image edits)	none (you can configure it if you want)	vllm_omni_service/
SGLang Diffusion	lmsysorg/sglang diffusion runtime	Bearer API key	sglang_diffusion_service/

Both methods share the same architecture: a FastAPI gateway accepts an image + prompt and immediately returns a task_id; a Celery worker pulls the job off a Redis broker, runs inference against the GPU engine, and writes the result to a shared storage/ volume. Clients submit a job, poll for status, then download the finished image.

Everything runs in Docker, orchestrated by each service's docker-compose.yml and driven by make commands.

serving_diffusion_models/
├── vllm_omni_service/              # Method 1 — vLLM Omni backend + gateway + worker
├── sglang_diffusion_service/       # Method 2 — SGLang diffusion backend + gateway + worker
├── vllm_client_tests/              # Async Python client for the vLLM gateway
├── sglang_diffusion_client_tests/  # Async Python client for the SGLang gateway
├── vllm_omni_frontend_test/        # React frontend for the vLLM gateway
└── test_input_images/              # Drop input images here for the client tests

Prerequisites

• Docker and an NVIDIA GPU with the NVIDIA Container Toolkit.
• A Hugging Face token with access to the model. This is optional but since some diffusion models are gated, setting your huggingface key is better.
• Python 3.10+ with httpx installed, to run the client test scripts (pip install httpx).

---

Method 1 — vLLM Omni

Endpoints (gateway on port 8000): POST /api/v1/edit-image · GET /api/v1/task-status/{task_id} · GET /files/{filename}

Run

cd vllm_omni_service
cp .env.example .env          # then set HF_TOKEN and MODEL_NAME
make build                    # build + start all services in the background

Useful targets: make up (start), make down (stop), make logs (follow logs), make status (list containers).

Test

Python client — point it at your gateway host and run it; it submits every image in test_input_images/, polls, and saves results to generated_images/:

pip install httpx
# edit API_BASE_URL in vllm_client_tests/test_async_single_image_vllm.py
cd vllm_client_tests
python test_async_single_image_vllm.py

There are two client scripts: test_async_single_image_vllm.py (single-image edits, one job per image) and test_async_multi_reference_vllm.py (multi-reference composition, see below).

Multiple reference images

The vLLM Omni gateway accepts one or more files under the file form field, so you can compose an edit from a base image plus reference images. The gateway persists every file and the worker compresses each to a data URL, passing them together to vLLM via extra_body["url"] — the engine's native multi-image mechanism (image=[] stays empty).

Send all files under the same file key (base image first) and address each one positionally in the prompt:

files = [
    ("file", ("room.png",       open("room.png", "rb"),       "image/png")),   # image 1 (base)
    ("file", ("chair.webp",     open("chair.webp", "rb"),     "image/webp")),  # image 2
    ("file", ("wallpaper.webp", open("wallpaper.webp", "rb"), "image/webp")),  # image 3
]
data = {"prompt": "Edit image 1 (the room). Place the chair from image 2 ... add nothing else."}

A ready-to-run client ships for this — edit the paths/prompt at the top of vllm_client_tests/test_async_multi_reference_vllm.py and run it:

cd vllm_client_tests
python test_async_multi_reference_vllm.py

Tips to avoid hallucinated content: keep it to 1–3 images total (the gateway rejects more than 3); reference images positionally ("image 1", "image 2"); tie each instruction to one input; and state what to preserve ("add nothing else") to suppress invented objects.

Frontend — see Method 1 — Frontend below.

---

Method 2 — SGLang Diffusion

Same async flow, plus a mandatory Bearer API key enforced by the gateway.

Endpoints (gateway on port 8000): POST /v1/images/edits · GET /v1/tasks/{task_id} · GET /files/{filename}

Run

cd sglang_diffusion_service
cp .env.example .env          # then set HF_TOKEN, SGLANG_MODEL, and SGLANG_API_KEY
make build

Same make targets as above (up, down, logs, status).

Test

pip install httpx
# in sglang_diffusion_client_tests/test_async_single_image_sglangdiffusion.py set:
#   API_BASE_URL  -> your gateway host
#   API_KEY       -> must match SGLANG_API_KEY in .env
cd sglang_diffusion_client_tests
python test_async_single_image_sglangdiffusion.py

Results are saved to sglang_generated_images/. There are two client scripts: test_async_single_image_sglangdiffusion.py (single-image edits, one job per image) and test_async_multi_reference_sglangdiffusion.py (multi-reference composition, see below).

Multiple reference images

The SGLang gateway accepts one or more files under the image form field, so you can compose an edit from a base image plus reference images (e.g. insert the chair from image 2 into the room in image 1). The gateway persists every file and the Celery worker forwards them all to the backend.

Send all files under the same image key (base image first), and address each one positionally in the prompt:

files = [
    ("image", ("room.png",       open("room.png", "rb"),       "image/png")),   # image 1 (base)
    ("image", ("chair.webp",     open("chair.webp", "rb"),     "image/webp")),  # image 2
    ("image", ("wallpaper.webp", open("wallpaper.webp", "rb"), "image/webp")),  # image 3
]
data = {
    "prompt": (
        "Edit image 1 (the room). Place the chair from image 2 into the room. "
        "Apply the wallpaper from image 3 to the walls. "
        "Keep the original layout, camera and lighting; add nothing else."
    ),
    "negative_prompt": "distortion, blur, low quality, extra furniture, deformed",
    "size": "1024x1024",
    "num_inference_steps": "40",
    "true_cfg_scale": "4.0",
}

A ready-to-run client ships for this — edit the paths/prompt at the top of sglang_diffusion_client_tests/test_async_multi_reference_sglangdiffusion.py and run it:

cd sglang_diffusion_client_tests
python test_async_multi_reference_sglangdiffusion.py

Tips to avoid hallucinated content:

• Keep it to 1–3 images total (base + up to 2 references). Quality degrades beyond ~3, and the gateway rejects more than 3.
• Reference images positionally — "image 1", "image 2" — and tie each instruction to one ("the chair from image 2", not just "add a chair").
• Be constraint-first: state what to preserve and "add nothing else" to suppress invented objects.
• Use a negative_prompt for failure modes, raise num_inference_steps to 40–50, and set a seed for reproducibility while tuning.

Supported form fields: image (one or more), prompt, negative_prompt, size, num_inference_steps, true_cfg_scale, seed, n, response_format.

---

Method 1 — Frontend

A React + TypeScript "digital darkroom" (vllm_omni_frontend_test/) that drives the vLLM Omni gateway: drop in images, write an edit prompt, watch each one develop through the async pipeline, and drag a slider to compare before/after.

cd vllm_omni_frontend_test
cp .env.example .env          # set VITE_API_TARGET to your gateway, e.g. http://<host>:8000
npm install
npm run dev                   # http://localhost:5173

The gateway sends no CORS headers, so the Vite dev server proxies all backend paths (see vite.config.ts). For a production build: npm run build && npm run preview.

---

Notes

• Each service writes inputs and generated images to its own storage/ directory (a mounted Docker volume). These are created at runtime and are not version-controlled. For production, you can set the storage backend to cloud storage like S3.
• Drop your own test images into test_input_images/; the folder is tracked but its contents are ignored.