Skip to content

feat: eval-containers on OpenShift via --cluster-config + GPU-served scenarios#1630

Open
natan-ergas wants to merge 7 commits into
llm-d:mainfrom
eladven:main
Open

feat: eval-containers on OpenShift via --cluster-config + GPU-served scenarios#1630
natan-ergas wants to merge 7 commits into
llm-d:mainfrom
eladven:main

Conversation

@natan-ergas

@natan-ergas natan-ergas commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

Enables the eval-containers agentic benchmarks (exgentic) to run on OpenShift
in two ways, and makes their scenarios cluster-portable:

  1. --cluster-config mechanism — eval-containers scenarios are now
    cluster-generic; cluster-specific values (RWX storage class, service account,
    runAsUser) are injected at run time from a --cluster-config file instead of
    being hardcoded in the scenario. Example cluster-configs ship under
    config/cluster-configs/examples/ (kind, openshift-generic,
    openshift-ibm-vpc, openshift-pokprod). The -kind scenario variants are
    removed in favor of this.

  2. GPU-served eval scenarios — new eval-containers-{gaia,aider-polyglot}-gpu
    scenarios that stand up a real vLLM model on GPUs (the standard llm-d
    modelservice path) and drive the agents against that in-cluster endpoint,
    rather than run-only against an external LiteLLM URL. Plus a GPU nightly
    workflow (reusable-ci-nightly-eval-gpu.yaml + gaia/polyglot callers) modeled
    on the perf nightly (standup → run → teardown).

Bundled framework fixes: poll-based pod wait (avoids a hang when short-lived
agentic pods finish before kubectl wait fires), conditional
dataAccess.runAsUser, stale data-access-pod delete-and-recreate, namespace
existence-check before apply, and harness.runAsUser/serviceAccount added to
the config schema.

Supersedes / continues the previously-closed #1627 (same fork→upstream
contribution). Opened as a draft for review.

Purpose

The eval-containers harness previously only ran run-only against IBM LiteLLM, and
its scenarios hardcoded IBM-cluster storage/SCC specifics — so they didn't port
to other clusters and couldn't exercise the standard llm-d "stand up a model and
serve it" path. This PR makes the scenarios portable (--cluster-config) and adds
a genuine GPU-served path so the agentic evals can run against a model llm-d
deploys, the same way the perf benchmarks do.

Intended follow-up: enter the gaia and aider-polyglot agentic runs
into the nightly benchmark (the GPU nightly workflow + callers in this PR are
the vehicle), so agentic quality is tracked nightly alongside the perf runs.

How it was tested

  • --cluster-config / plumbing: aider-polyglot ran end-to-end on an OpenShift
    namespace (5/5 pods completed, results collected, clean teardown) via
    --cluster-config; kind + OpenShift render/plan validated.
  • GPU-served path (pok-prod, 14× H100): both scenarios run end-to-end,
    standup → run → teardown, serving Qwen/Qwen3-32B:
    • Model auto-discovered via the in-cluster epponly endpoint (no -U, no key).
    • 30-task runs at decode.replicas: 10 (10 GPUs, load-balanced): all 30
      harness pods complete, 30/30 in-pod gateways start, results collected +
      analyzed. gaia and aider-polyglot both exercised.
    • Scores are per-task reward/passed in each task/result.json (the harness
      does not compute an aggregate; pass-rate is tallied from the per-task files).
      Current scores reflect the served model / benchmark difficulty, not pipeline
      defects (gaia needs a working web-search tool; aider-polyglot is blind
      single-shot coding).
  • Render validation + detect-secrets pre-commit hooks pass on the changed
    files. (pytest suite unaffected — the GPU changes are scenario YAML + one
    template line + workflow YAML, no Python.)

Notes for reviewers

  • Two model/agent lessons baked into the GPU scenarios: serve Qwen3-32B
    (gpt-oss-120b is incompatible with both agents — gemini-cli's JSON router can't
    parse its harmony tool calls; codex's web_search tool is rejected by
    vLLM-served gpt-oss), and decode.replicas must track -j (~2–3 tasks/GPU)
    or agents starve on the per-task timeout.
  • 04_download_job.yaml.j2: pip install -U --user huggingface_hub so the model
    download job has the hf CLI even on exgentic images (which ship
    huggingface_hub 0.26.2, pre-hf-CLI). Longer-term this belongs in the
    exgentic image build (huggingface_hub>=0.34).

Authored by @eladven (the --cluster-config feature, tenant fixes, docs) and
@natan-ergas (GPU-served scenarios + nightly). All commits DCO signed-off.

@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

@natan-ergas
natan-ergas force-pushed the main branch 4 times, most recently from 5f36029 to c1bd2dc Compare July 13, 2026 22:05
natan-ergas added a commit to eladven/llm-d-benchmark that referenced this pull request Jul 14, 2026
@natan-ergas @eladven
feat: eval-containers on OpenShift via --cluster-config + GPU-served …
a4c2dd9 
…scenarios

Squash of the eval-containers OpenShift/GPU work (see PR llm-d#1630).

--cluster-config mechanism: eval-containers scenarios are cluster-generic;
RWX storage class, service account, and runAsUser come from a --cluster-config
file merged at run time. Example cluster-configs under
config/cluster-configs/examples/ (kind, openshift-generic, openshift-ibm-vpc,
openshift-pokprod). Bundled framework fixes: poll-based pod wait, conditional
dataAccess.runAsUser (default null, opt-in root), stale data-access-pod
delete-and-recreate, namespace existence-check, harness.runAsUser/serviceAccount
in the config schema.

GPU-served scenarios: eval-containers-{gaia,aider-polyglot}-gpu stand up a real
vLLM model on GPUs (modelservice) and drive the agents against the in-cluster
endpoint (no -U/-g). decode.replicas scales serving capacity; modelCommand:
custom so --max-model-len reaches vLLM. GPU nightly workflow
(reusable-ci-nightly-eval-gpu.yaml + gaia/polyglot callers). Serves Qwen3-32B.
04_download_job.yaml.j2: pip install -U --user huggingface_hub so the download
job has the 'hf' CLI on exgentic images. docs/agentic_eval.md rewritten.

Co-authored-by: Elad Venezian <eladv@il.ibm.com>
Signed-off-by: Natan Ergas <natan.davids@mail.huji.ac.il>
natan-ergas added a commit to eladven/llm-d-benchmark that referenced this pull request Jul 14, 2026
@natan-ergas
feat: eval-containers on OpenShift via --cluster-config + GPU-served …
13cfd2f 
…scenarios

Squash of the eval-containers OpenShift/GPU work (see PR llm-d#1630).

--cluster-config mechanism: eval-containers scenarios are cluster-generic;
RWX storage class, service account, and runAsUser come from a --cluster-config
file merged at run time. Example cluster-configs under
config/cluster-configs/examples/ (kind, openshift-generic, openshift-ibm-vpc,
openshift-pokprod). Bundled framework fixes: poll-based pod wait, conditional
dataAccess.runAsUser (default null, opt-in root), stale data-access-pod
delete-and-recreate, namespace existence-check, harness.runAsUser/serviceAccount
in the config schema.

GPU-served scenarios: eval-containers-{gaia,aider-polyglot}-gpu stand up a real
vLLM model on GPUs (modelservice) and drive the agents against the in-cluster
endpoint (no -U/-g). decode.replicas scales serving capacity; modelCommand:
custom so --max-model-len reaches vLLM. GPU nightly workflow
(reusable-ci-nightly-eval-gpu.yaml + gaia/polyglot callers). Serves Qwen3-32B.
04_download_job.yaml.j2: pip install -U --user huggingface_hub so the download
job has the 'hf' CLI on exgentic images. docs/agentic_eval.md rewritten.

Signed-off-by: Natan Ergas <natan.davids@mail.huji.ac.il>
@natan-ergas
feat: eval-containers on OpenShift via --cluster-config + GPU-served …
af7596b 
…scenarios

Squash of the eval-containers OpenShift/GPU work (see PR llm-d#1630).

--cluster-config mechanism: eval-containers scenarios are cluster-generic;
RWX storage class, service account, and runAsUser come from a --cluster-config
file merged at run time. Example cluster-configs under
config/cluster-configs/examples/ (kind, openshift-generic, openshift-ibm-vpc,
openshift-pokprod). Bundled framework fixes: poll-based pod wait, conditional
dataAccess.runAsUser (default null, opt-in root), stale data-access-pod
delete-and-recreate, namespace existence-check, harness.runAsUser/serviceAccount
in the config schema.

GPU-served scenarios: eval-containers-{gaia,aider-polyglot}-gpu stand up a real
vLLM model on GPUs (modelservice) and drive the agents against the in-cluster
endpoint (no -U/-g). decode.replicas scales serving capacity; modelCommand:
custom so --max-model-len reaches vLLM. GPU nightly workflow
(reusable-ci-nightly-eval-gpu.yaml + gaia/polyglot callers). Serves Qwen3-32B.
04_download_job.yaml.j2: pip install -U --user huggingface_hub so the download
job has the 'hf' CLI on exgentic images. docs/agentic_eval.md rewritten.

Signed-off-by: Natan Ergas <natan.davids@mail.huji.ac.il>
@alonh
alonh force-pushed the main branch 2 times, most recently from 9ddf42c to 2f46edb Compare July 16, 2026 07:48
@alonh
GPU eval scenarios: serve Qwen3.6-27B with qwen3_coder tool parser
2091d5e 
Switch both eval-containers GPU scenarios (gaia, aider-polyglot) from
Qwen/Qwen3-32B to Qwen/Qwen3.6-27B, and update the vLLM tool-call parser
from hermes to qwen3_coder to match the newer model.

On aider-polyglot (gemini-cli, 100 tasks, 10 GPUs) this raises the pass
rate to 23/100 from 5/100 with Qwen3-32B.

Signed-off-by: Alon Halfon <alon.halfon@gmail.com>
@alonh
Remove gaia from the GPU nightly until web_search is fixed
a9387fb 
The gaia/codex GPU nightly caller is dropped because the codex agent's
web_search tool returns 'unsupported call' in the exgentic image, so gaia
scores 0 regardless of model — not worth a nightly run until that is
resolved. Keep the aider-polyglot/gemini-cli nightly, and drop the stale
02:17 gaia offset reference from its schedule comment. The gaia-gpu
scenario itself is retained for manual runs.

Signed-off-by: Alon Halfon <alon.halfon@gmail.com>
@natan-ergas
natan-ergas marked this pull request as ready for review July 16, 2026 14:45
@alonh
eval-gpu nightly: workflow_dispatch-only, drop cron schedule
260869f 
Per the repo's nightly pattern, the cron schedule lives in the llm-d
orchestrator repo (which cross-dispatches into this workflow), not here.
Make the aider-polyglot eval-gpu workflow dispatch-only so a follow-up PR
against llm-d can own the schedule once this workflow is proven stable.

Signed-off-by: Alon Halfon <alon.halfon@gmail.com>
@mengmeiye

mengmeiye commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Hi @alonh @natan-ergas Could you please fix the CI errors? Thank you.

@alonh
Apply ruff lint/format and fix broken README doc anchors
e05a6d1 
Run the repo's ruff-check --fix + ruff-format (v0.15.11, matching the
pre-commit hooks) over the changed Python files; this splits the
semicolon-crammed poll loop in kube_helpers.py and wraps long calls.
No logic changes. Also fix two broken anchor links in docs/agentic_eval.md
(#verified-benchmark--agent-combinations and #troubleshooting).

Signed-off-by: Alon Halfon <alon.halfon@gmail.com>
@alonh

alonh commented Jul 19, 2026

Copy link
Copy Markdown

Hi @alonh @natan-ergas Could you please fix the CI errors? Thank you.

Fixed. Thanks for the quick review!

alonh added 2 commits July 19, 2026 20:06
@alonh
eval-gpu polyglot: set EVAL_TIMEOUT=1500 for the agent wall-clock
ffb9428 
The exgentic image's default per-task agent timeout (300s) was too tight
for Qwen3.6-27B, killing many tasks mid-edit. Propagate EVAL_TIMEOUT=1500
via harness.extraEnvVars. A 100-task run at 1500 saw a max runtime of
~1020s with 0 tasks hitting the ceiling, and the pass rate rose 23 -> 28.

Signed-off-by: Alon Halfon <alon.halfon@gmail.com>
@alonh
Fix agentic_eval anchor: restore double-hyphen for the × in the heading
6118c7f 
The '## Verified benchmark × agent combinations' heading slugifies to
verified-benchmark--agent-combinations (the × collapses, leaving two
hyphens from the surrounding spaces). An earlier change wrongly used a
single hyphen, which markdown-link-check flags as a 404. Verified with
the CI checker (markdown-link-check + .markdown-link-check.json): the
single-hyphen form fails, the double-hyphen form passes (10/10 links).

Signed-off-by: Alon Halfon <alon.halfon@gmail.com>
@mengmeiye

mengmeiye commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Hi @alonh @natan-ergas Thanks for the fix. The code looks good to me. Before we approve and merge it, could you please rebase your PR to upstream? Thanks again.

Vezio
Vezio requested changes Jul 24, 2026
View reviewed changes
Comment thread llmdbenchmark/cli.py
help="Manual OIDC token or API key for telemetry auth.",
)

benchmark_parser.add_argument(

@Vezio Vezio Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is cluster_config used within llm-d-benchmark?

I see that - for example - config/cluster-configs/examples/openshift-pokprod.yaml is consumed as input to the parser, can't this information be stored the scenario itself? I believe we support these fields - or similar ones today - it would be a better way to consolidate and allow users to reproduce workloads by having it in one spot, rather than merged in under the covers.

What is the benefit from separating it from the scenario and what options are supported there?

If it is more usable to have it separated, we should pursue that - but I want to avoid many different ways to changes these knobs since that will become a maintenance burden.

Thanks team!

@Vezio Vezio Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CC: @mengmeiye what are your thoughts here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Vezio Vezio Vezio requested changes
@maugustosilva maugustosilva Awaiting requested review from maugustosilva maugustosilva is a code owner
@achandrasekar achandrasekar Awaiting requested review from achandrasekar achandrasekar is a code owner
@namasl namasl Awaiting requested review from namasl namasl is a code owner
@mengmeiye mengmeiye Awaiting requested review from mengmeiye mengmeiye is a code owner
@kalantar kalantar Awaiting requested review from kalantar kalantar is a code owner

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@natan-ergas @mengmeiye @alonh @Vezio