diff --git a/Docs/harbor_fork_requirements.md b/Docs/harbor_fork_requirements.md new file mode 100644 index 0000000..79bfb3d --- /dev/null +++ b/Docs/harbor_fork_requirements.md @@ -0,0 +1,187 @@ +# Harbor Fork — Integration Requirements for ABEvalFlow + +> **Target repo:** [RHEcosystemAppEng/skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) +> **Open PR:** [#1 — feat: add OpenShift environment backend](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/1) +> **ABEvalFlow branch:** `APPENG-4906/harbor-eval-task` + +--- + +## ABEvalFlow-Side Implementation (current state) + +### How ABEvalFlow invokes Harbor + +The `harbor-eval` Tekton task (`pipeline/tasks/harbor-eval.yaml`) runs a single +step that: + +1. Installs Harbor from the fork via `pip install git+@` +2. Generates **two separate Harbor job configs** (one per variant) using + `scripts/generate_eval_config.py` +3. Runs `harbor run -c treatment-config.yaml` followed by + `harbor run -c control-config.yaml` +4. Parses `result.json` files from each variant's results directory to compute + pass rates + +### Per-variant config structure + +Each config is a standard Harbor `JobConfig` YAML with a **single task** and +the image ref set via global `environment.kwargs.image_ref`: + +```yaml +# treatment-config.yaml +job_name: my-submission-treatment +jobs_dir: /workspace/eval-results/my-submission/treatment +n_attempts: 20 +environment: + type: openshift + delete: true + kwargs: + image_ref: "registry/ns/my-submission@sha256:abc..." + override_cpus: 1 + override_memory_mb: 2048 + override_storage_mb: 10240 +agents: + - {} +tasks: + - path: /workspace/tasks-treatment/my-submission +``` + +Control config is identical but with the control image ref and task path. + +### Result directory layout + +``` +eval-results// + treatment/ + / + __/result.json + __/result.json + ... (N trials) + control/ + / + __/result.json + ... (N trials) +``` + +### What ABEvalFlow reads from metadata.yaml + +The config generator extracts these fields from `SubmissionMetadata`: + +| Field | Maps to | Default | +|-------|---------|---------| +| `experiment.n_trials` | `n_attempts` | 20 | +| `agent_timeout_sec` | `agent_timeout_multiplier` (ratio vs 600s) | 1.0x | +| `verifier_timeout_sec` | `verifier_timeout_multiplier` (ratio vs 120s) | 1.0x | +| `agent_setup_timeout_sec` | `agent_setup_timeout_multiplier` (ratio vs 600s) | 1.0x | +| `build_timeout_sec` | `environment_build_timeout_multiplier` (ratio vs 600s) | 1.0x | +| `cpus` | `environment.override_cpus` | 1 | +| `memory_mb` | `environment.override_memory_mb` | 2048 | +| `storage_mb` | `environment.override_storage_mb` | 10240 | + +### Eval modes + +| Mode | `--ek image_ref` | Image source | When to use | +|------|------------------|--------------|-------------| +| `prebuilt` | Set to digest ref from build-push task | Tekton builds with Buildah, pushes to internal registry | Default pipeline flow | +| `local-build` | Not set; `force_build: true` | Harbor builds from `environment/Dockerfile` in each task dir | Local dev, or when skipping the build-push step | + +### Tekton results emitted + +| Result | Description | +|--------|-------------| +| `treatment-pass-rate` | Decimal string (e.g. `"0.8500"`) | +| `control-pass-rate` | Decimal string (e.g. `"0.6000"`) | +| `results-dir` | Absolute path to the results base directory | + +--- + +## What the Harbor fork must support + +### Required (blocking) + +**1. `harbor run -c ` with OpenShift environment** + +The fork's `OpenShiftEnvironment` must handle the full trial lifecycle when +invoked with `--env openshift` (or `environment.type: openshift` in config): + +- Accept `image_ref` via `environment.kwargs` — verify the image is pullable, + skip building +- Create trial Pods with the pre-built image +- Execute agent + verifier inside the Pod via `exec` +- Upload/download files via tar-over-exec +- Write `result.json` with `verifier_result.reward` (float: 1.0 = pass, 0.0 = fail) +- Clean up Pods after each trial (`delete: true`) + +**Status:** Implemented in PR #1. Needs merge. + +**2. `environment.kwargs` passthrough in config-based invocation** + +When `harbor run -c config.yaml` is used, the `environment.kwargs` dict from +the config must be passed to the environment's `__init__`. This is how +`image_ref` reaches `OpenShiftEnvironment`. + +**Status:** Verify this works in `harbor jobs start` with YAML config +(vs CLI `--ek` flags). The CLI path (`--ek image_ref=...`) is tested; +the config path should behave identically but needs confirmation. + +**3. `result.json` output format** + +ABEvalFlow's pass-rate parser expects: + +```json +{ + "verifier_result": { + "reward": 1.0 + } +} +``` + +Where `reward > 0.0` means pass. This is Harbor's standard format — no change +needed, but any deviation would break the parser. + +### Implemented (fork PR #2) + +**4. Per-task `environment_kwargs` support** + +Implemented in [PR #2](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/2). +`TaskConfig` now has an `environment_kwargs: dict[str, Any]` field. `Job._env_config_for_task` +merges per-task kwargs into the global `EnvironmentConfig.kwargs` (task-level overrides global). + +ABEvalFlow currently runs each variant as a separate Harbor job, which works +without this feature. Per-task kwargs enables a single-job alternative for +sweep-based workflows: + +```yaml +# Single-job example (supported since PR #2): +tasks: + - path: /workspace/tasks-treatment/my-submission + environment_kwargs: + image_ref: "registry/ns/my-submission@sha256:abc..." + - path: /workspace/tasks-control/my-submission + environment_kwargs: + image_ref: "registry/ns/my-submission@sha256:def..." +``` + +--- + +## Handoff Doc Alignment (WS3A) + +The existing handoff doc (`Docs/harbor_openshift_backend.md`) has diverged from +the actual implementation in PR #1. These items should be updated: + +| Section | Current (outdated) | Correct | +|---------|-------------------|---------| +| File path | `openshift_environment.py` | `openshift.py` | +| Build modes | `_build_and_push_image` is no-op only | Supports pre-built (`--ek image_ref=`) AND local podman build | +| Pod security | `readOnlyRootFilesystem: true` | Intentionally unset — many agent workloads need writes; `HOME=/tmp` is injected instead | +| RBAC table | ConfigMaps, Secrets, PVCs, ImageStreams | Only Pods + exec + Secrets used in practice | +| Naming | "skilled / unskilled" | "treatment / control" | +| Trial count | "20 skilled + 20 unskilled" | "20 treatment + 20 control" | +| Tekton params | `skilled-image-ref` / `unskilled-image-ref` | `treatment-image-ref` / `control-image-ref` | +| Definition of Done | "20 skilled + 20 unskilled" | "20 treatment + 20 control" | + +### Additional Notes + +- The OpenShift backend supports a `cpu_request` kwarg (`--ek cpu_request=`) + for clusters with tight resource constraints — not documented in the handoff doc. +- The `--ek registry=` kwarg enables local podman build+push to a specified + registry — also undocumented. diff --git a/Docs/harbor_openshift_backend.md b/Docs/harbor_openshift_backend.md index ccbf190..8f60dca 100644 --- a/Docs/harbor_openshift_backend.md +++ b/Docs/harbor_openshift_backend.md @@ -1,75 +1,114 @@ # Harbor OpenShift Backend — Handoff Document > **Jira:** APPENG-4906 (Phase 4 — Harbor OpenShift Backend) -> **Target repo:** [GuyZivRH/skills_eval_corrections](https://github.com/GuyZivRH/skills_eval_corrections) (Harbor fork) -> **Full spec:** See [implementation_plan.md](./implementation_plan.md), Phase 4 (lines 270–343) +> **Target repo:** [RHEcosystemAppEng/skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) (Harbor fork) +> **Fork PRs:** [#1 — OpenShift environment backend](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/1), [#2 — per-task environment_kwargs](https://github.com/RHEcosystemAppEng/skills_eval_corrections/pull/2) +> **Full spec:** See [implementation_plan.md](./implementation_plan.md), Phase 4 --- -## What to Build +## What Was Built -A new `OpenShiftEnvironment` class in the Harbor fork that enables `harbor run --env openshift`. This backend manages trial Pod lifecycle on OpenShift using pre-built container images (built by ABEvalFlow's Tekton pipeline). +An `OpenShiftEnvironment` class in the Harbor fork that enables +`harbor run --env openshift`. This backend manages trial Pod lifecycle +on OpenShift and supports two modes: pre-built container images (from +the Tekton pipeline) and local build via podman. ## Where in the Harbor Fork | File | Action | |---|---| -| `src/harbor/environments/openshift_environment.py` | Create — new backend | -| `src/harbor/models/environment_type.py` | Edit — add `OPENSHIFT = "openshift"` to enum | -| Backend registration (entry point or factory) | Edit — register so `--env openshift` works | -| `tests/` | Create — unit tests with mocked K8s API | +| `src/harbor/environments/openshift.py` | New backend | +| `src/harbor/environments/k8s_client_manager.py` | Shared K8s client management (independent instances per caller) | +| `src/harbor/models/environment_type.py` | `OPENSHIFT = "openshift"` added to enum | +| `src/harbor/models/trial/config.py` | `environment_kwargs` field added to `TaskConfig` (PR #2) | +| `src/harbor/job.py` | `_env_config_for_task` merges per-task kwargs into env config (PR #2) | +| `tests/unit/environments/test_openshift.py` | Unit tests with mocked K8s API | +| `tests/unit/models/test_trial_task_config.py` | Tests for per-task kwargs merge behavior | +| `examples/configs/openshift-*.yaml` | Config examples for both modes + per-task kwargs | ## Reference Implementation -Use the GKE backend as your template: `src/harbor/environments/gke.py` (~1044 lines). It implements the full `BaseEnvironment` interface from `src/harbor/environments/base.py`. +The GKE backend (`src/harbor/environments/gke.py`) implements the full +`BaseEnvironment` interface from `src/harbor/environments/base.py`. ## Key Differences from GKE ### `_init_client` - GKE: `gcloud container clusters get-credentials` + `load_kube_config()` -- **OpenShift:** `load_incluster_config()` when running inside the cluster (Tekton), or `load_kube_config()` for local dev +- **OpenShift:** `load_incluster_config()` when running inside the cluster + (Tekton), falls back to `load_kube_config()` for local dev ### `_build_and_push_image` - GKE: `gcloud builds submit` (Cloud Build) -- **OpenShift: No-op.** Image build/push is handled by Tekton (Phase 3). The backend receives a digest-based image reference and only verifies the image exists and is pullable. +- **OpenShift — two modes:** -**Contract:** +| Mode | Behavior | When to use | +|------|----------|-------------| +| **Prebuilt** (`image_ref` kwarg set) | Verify image is pullable, skip building | Default pipeline flow — Tekton builds with Buildah | +| **Local build** (`force_build: true`) | Build with podman, push to specified registry | Local dev, skipping the build-push Tekton step | + +**Prebuilt contract:** - **Input:** Immutable image ref (e.g., `image-registry.openshift-image-registry.svc:5000/ab-eval-flow/my-skill@sha256:abc...`) - **Behavior:** Verify image exists using the trial ServiceAccount - **Output:** Return the image ref unchanged -- **Failure:** Raise `ImageNotFoundError` / `ImageNotPullableError` with the image ref and SA identity +- **Failure:** Raise `ImageNotFoundError` / `ImageNotPullableError` -### `_image_exists` -- GKE: `gcloud artifacts docker images describe` -- **OpenShift:** Query internal registry API or use `skopeo inspect` +**Local build contract:** +- **Input:** `environment/Dockerfile` in the task directory, `registry` kwarg for push target +- **Behavior:** `podman build` + `podman push` to the specified registry +- **Kwargs:** `registry` (push target URL), `tls_verify` (default `"true"`) ### `start`, `stop`, `exec`, `upload_file/dir`, `download_file/dir` - Same K8s API patterns as GKE — the `kubernetes` Python client is identical -## Pod Security Requirements +## Environment kwargs + +Kwargs are passed via `environment.kwargs` in the job config YAML or +via `--ek key=value` on the CLI. Per-task kwargs (PR #2) override +global kwargs when set. + +| Kwarg | Description | Default | +|-------|-------------|---------| +| `namespace` | OpenShift namespace for trial Pods | Required | +| `image_ref` | Digest-based image ref (prebuilt mode) | — | +| `registry` | Push target URL (local-build mode) | — | +| `cpu_request` | CPU request for trial Pods (e.g. `"250m"`) | — | +| `tls_verify` | TLS verification for registry (e.g. `"false"`) | `"true"` | + +### Per-task environment_kwargs (PR #2) -Trial Pods must run with hardened security context: +`TaskConfig` now supports an `environment_kwargs` dict that merges into +the global `environment.kwargs` (task-level overrides global). This +enables treatment and control variants with different image refs in a +single Harbor job: ```yaml -securityContext: - runAsNonRoot: true - readOnlyRootFilesystem: true - allowPrivilegeEscalation: false - capabilities: - drop: ["ALL"] - seccompProfile: - type: RuntimeDefault +tasks: + - path: /workspace/tasks-treatment/my-submission + environment_kwargs: + image_ref: "registry/ns/my-submission@sha256:abc..." + - path: /workspace/tasks-control/my-submission + environment_kwargs: + image_ref: "registry/ns/my-submission@sha256:def..." ``` -Since `readOnlyRootFilesystem: true` prevents writes, mount `emptyDir` volumes for: -- `/tmp` -- Agent cache directories (varies by agent) +ABEvalFlow currently runs each variant as a separate Harbor job (two +`harbor run` invocations). Per-task kwargs enables a single-job +alternative if needed in the future. + +## Pod Security Requirements + +Trial Pods run with OpenShift's default security constraints. The +`readOnlyRootFilesystem` is intentionally **not** set — many agent +workloads need filesystem writes. Instead, `HOME=/tmp` is injected to +direct writes to a writable location. Verify the target cluster uses `restricted-v2` SCC (OpenShift 4.11+). ## Trial Execution -- **N = 20** attempts per variant (skilled + unskilled = 40 total sessions) +- **N = 20** attempts per variant (treatment + control = 40 total sessions) - Image refs come as params from the build-push Tekton task (digest-based) - LLM endpoint via environment variable — backend is agnostic to LLM access mode - Configurable per-trial timeout and global evaluation timeout @@ -82,37 +121,52 @@ The pipeline ServiceAccount in `ab-eval-flow` namespace needs: | Resource | Verbs | Purpose | |---|---|---| | Pods | create, get, list, watch, delete | Trial Pod lifecycle | -| ConfigMaps | get, list | Trial configuration | +| Pods/exec | create | Agent/verifier execution inside trial Pods | +| Pods/log | get | Retrieving trial output | | Secrets | get | LLM credentials injection | | Events | get, list | Diagnosing hung/failed Pods | -| PVCs | get, list, create | Pipeline workspaces | -| ImageStreams | get, list | Registry access | + +## K8s Client Management + +The `BaseK8sClientManager` (shared by GKE and OpenShift backends) was +refactored in PR #2 to return **independent `CoreV1Api` instances** per +caller, each backed by its own `ApiClient`. This prevents concurrent +`kubernetes.stream.stream()` calls (which monkey-patch `ApiClient.call_api`) +from interfering with regular REST calls in other coroutines. ## Testing Strategy - **Unit tests:** Mock K8s API with `pytest` + `unittest.mock`. No live cluster needed. -- **Integration tests:** Test against OpenShift developer sandbox (ROSA/OSD). Do **not** use Kind/Minikube — they won't catch SCC/Routes differences. +- **Integration tests:** Test against OpenShift developer sandbox (ROSA/OSD). + Do **not** use Kind/Minikube — they won't catch SCC/Routes differences. -## Tekton Task (in ABEvalFlow repo, not Harbor fork) +## Tekton Task (in ABEvalFlow repo) -A `pipeline/tasks/harbor-eval.yaml` Tekton Task will also be needed in ABEvalFlow to invoke Harbor. This task: -- Accepts `skilled-image-ref` and `unskilled-image-ref` as params -- Runs `harbor run --env openshift` with the image refs -- Collects results to workspace/PVC +The `harbor-eval` Tekton task (`pipeline/tasks/harbor-eval.yaml`) invokes +Harbor from the ABEvalFlow pipeline: -This can be built after the backend is functional. +- Installs Harbor from the fork via `pip install git+@` +- Generates two per-variant job configs using `scripts/generate_eval_config.py` +- Runs `harbor run -c treatment-config.yaml` then `harbor run -c control-config.yaml` +- Parses `result.json` files to compute pass rates as Tekton results +- Supports `prebuilt` and `local-build` eval modes via a pipeline param ## Definition of Done -- [ ] 40 trial Pods complete (20 skilled + 20 unskilled) +- [x] `OpenShiftEnvironment` backend implemented (PR #1) +- [x] Per-task `environment_kwargs` support (PR #2) +- [x] Independent K8s client instances per caller (PR #2) +- [x] Config examples for prebuilt, local-build, and per-task modes (PR #2) +- [x] Unit tests with mocked K8s API +- [ ] 40 trial Pods complete (20 treatment + 20 control) on live cluster - [ ] Cleanup verified — no stale Pods after evaluation - [ ] Retry behavior validated for transient failures -- [ ] Unit tests pass with mocked K8s API - [ ] Integration test passes on OpenShift sandbox ## LLM Access Modes (for reference) -The backend doesn't need to know which mode is used — it just passes env vars to trial Pods: +The backend doesn't need to know which mode is used — it just passes +env vars to trial Pods: | Mode | Env Var | Infrastructure | |---|---|---| diff --git a/Docs/implementation_plan.md b/Docs/implementation_plan.md index 06c8d7a..efc4f2d 100644 --- a/Docs/implementation_plan.md +++ b/Docs/implementation_plan.md @@ -26,7 +26,7 @@ The `tasks-treatment/` and `tasks-control/` directories generated during scaffol ### Harbor Fork -The Harbor fork is at [GuyZivRH/skills_eval_corrections](https://github.com/GuyZivRH/skills_eval_corrections), forked from `harbor-framework/harbor`. The OpenShift backend will be added here as a new environment type alongside the existing GKE, Docker, Podman, Daytona, Modal, etc. +The Harbor fork is at [RHEcosystemAppEng/skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections), forked from `harbor-framework/harbor`. The OpenShift backend will be added here as a new environment type alongside the existing GKE, Docker, Podman, Daytona, Modal, etc. ### LLM Access Strategy (Updated from ADR) @@ -104,7 +104,7 @@ ABEvalFlow/ | EventListener route/URL | Webhook target | Exposed via OpenShift Route; URL needed for GitHub webhook setup | | Submissions repo deploy key (read) | Clone submission on pipeline trigger | Stored as OpenShift Secret | | Quay.io registry | Image storage | Service account + push secret | -| Harbor fork | Evaluation engine | [GuyZivRH/skills_eval_corrections](https://github.com/GuyZivRH/skills_eval_corrections) with OpenShift backend | +| Harbor fork | Evaluation engine | [RHEcosystemAppEng/skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) with OpenShift backend | | LLM access | Agent inference | One of: direct API key, opencode+self-hosted, or Vertex AI+LiteLLM | | PVC or MinIO (S3) | Artifact storage | For reports, logs, images | | agentic-collections deploy key (write) | Publish passing skills | Token/deploy key stored as OpenShift Secret | @@ -269,7 +269,7 @@ Use digest-based references (not mutable tags) between tasks to avoid tag mutati ### 4.1 OpenShift Environment Backend (in Harbor fork) -**Goal:** Create a new `OpenShiftEnvironment` class extending `BaseEnvironment` (from `src/harbor/environments/base.py`) in the [Harbor fork](https://github.com/GuyZivRH/skills_eval_corrections). +**Goal:** Create a new `OpenShiftEnvironment` class extending `BaseEnvironment` (from `src/harbor/environments/base.py`) in the [Harbor fork](https://github.com/RHEcosystemAppEng/skills_eval_corrections). The GKE backend (`src/harbor/environments/gke.py`, ~1044 lines) serves as the reference implementation. The OpenShift backend must implement the full `BaseEnvironment` interface: @@ -300,10 +300,11 @@ Additional requirements: ### 4.2 Harbor Fork Integration -- [ ] Add `openshift_environment.py` in `src/harbor/environments/`. -- [ ] Add `OPENSHIFT` to `EnvironmentType` enum. -- [ ] Register the backend so `harbor run --env openshift` selects it. +- [x] Add `openshift.py` in `src/harbor/environments/` (PR #1 in fork). +- [x] Add `OPENSHIFT` to `EnvironmentType` enum (PR #1 in fork). +- [x] Register the backend so `harbor run --env openshift` selects it (PR #1 in fork). - [ ] Pin a fork SHA in the pipeline image; review upstream quarterly for drift. +- [ ] Add per-task `environment_kwargs` to `TaskConfig` (see `Docs/harbor_fork_requirements.md`). ### 4.3 Testing Strategy @@ -312,24 +313,28 @@ Additional requirements: ### 4.4 Trial Execution Configuration -- The `harbor-eval` Tekton task accepts `treatment-image-ref` and `control-image-ref` as **params** wired from Phase 3 results. -- N = configurable attempts per variant (default 20, treatment + control = 40 total sessions). -- Configure resource requests/limits per trial Pod. -- LLM endpoint configured via environment variable — backend is agnostic to whether it points to LiteLLM, a direct API, or a self-hosted model. -- Trial Pod timeout: configurable, with a global evaluation timeout. +- [x] `harbor-eval` Tekton task accepts `treatment-image-ref` and `control-image-ref` as params wired from Phase 3 results. +- [x] N = configurable attempts per variant (default 20, treatment + control = 40 total sessions) via `n-trials` from `metadata.yaml`. +- [x] Resource requests/limits per trial Pod (from `metadata.yaml`: `cpus`, `memory_mb`, `storage_mb`). +- [ ] LLM endpoint configured via environment variable — backend is agnostic to whether it points to LiteLLM, a direct API, or a self-hosted model. +- [x] Trial Pod timeout: configurable via timeout multipliers derived from `metadata.yaml`. +- [x] Eval config generation script (`scripts/generate_eval_config.py`) reads metadata and produces Harbor job config YAML. +- [x] Supports two modes: `prebuilt` (digest image refs) and `local-build` (Harbor builds from Dockerfiles). ### 4.5 RBAC Requirements +- [x] `pipeline-trial-manager` Role + RoleBinding added to `config/rbac.yaml`. + The pipeline ServiceAccount needs (prefer named Secrets for least-privilege where policy requires): -| Resource | Verbs | Purpose | -|---|---|---| -| Pods | create, get, list, watch, delete | Trial Pod lifecycle (watch for efficient state tracking) | -| ConfigMaps | get, list | Trial configuration | -| Secrets | get | LLM credentials injection via `envFrom` | -| Events | get, list | Diagnosing hung/failed trial Pods | -| PVCs | get, list, create | Pipeline workspaces and artifacts | -| ImageStreams (OpenShift) | get, list | Registry access — validate against target cluster's API version | +| Resource | Verbs | Purpose | Status | +|---|---|---|---| +| Pods, Pods/exec, Pods/log | create, get, list, watch, delete | Trial Pod lifecycle | Done | +| Secrets | get | LLM credentials injection via `envFrom` | Done | +| Events | get, list | Diagnosing hung/failed trial Pods | Done | +| ConfigMaps | get, list | Trial configuration | Deferred — not used by current backend | +| PVCs | get, list, create | Pipeline workspaces and artifacts | Deferred — handled by Tekton | +| ImageStreams (OpenShift) | get, list | Registry access | Deferred — not used by current backend | ### 4.6 Definition of Done diff --git a/README.md b/README.md index b3ba423..ae5894b 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ ABEvalFlow/ | Repository | Purpose | |---|---| | [agentic-collections](https://github.com/RHEcosystemAppEng/agentic-collections) | Production skills, committed after evaluation passes | -| [skills_eval_corrections](https://github.com/GuyZivRH/skills_eval_corrections) | Harbor fork with OpenShift backend | +| [skills_eval_corrections](https://github.com/RHEcosystemAppEng/skills_eval_corrections) | Harbor fork with OpenShift backend | ## Submission Contract diff --git a/config/rbac.yaml b/config/rbac.yaml index e500c63..0b313e8 100644 --- a/config/rbac.yaml +++ b/config/rbac.yaml @@ -11,3 +11,39 @@ subjects: - kind: ServiceAccount name: pipeline namespace: ab-eval-flow +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: pipeline-trial-manager + namespace: ab-eval-flow +rules: + - apiGroups: [""] + resources: [pods] + verbs: [create, get, list, watch, delete] + - apiGroups: [""] + resources: [pods/exec] + verbs: [create] + - apiGroups: [""] + resources: [pods/log] + verbs: [get] + - apiGroups: [""] + resources: [events] + verbs: [get, list] + - apiGroups: [""] + resources: [secrets] + verbs: [get] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: pipeline-trial-manager + namespace: ab-eval-flow +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: pipeline-trial-manager +subjects: + - kind: ServiceAccount + name: pipeline + namespace: ab-eval-flow diff --git a/pipeline/tasks/harbor-eval.yaml b/pipeline/tasks/harbor-eval.yaml new file mode 100644 index 0000000..c061f97 --- /dev/null +++ b/pipeline/tasks/harbor-eval.yaml @@ -0,0 +1,183 @@ +apiVersion: tekton.dev/v1 +kind: Task +metadata: + name: harbor-eval + namespace: ab-eval-flow +spec: + description: >- + Runs A/B evaluation using Harbor on OpenShift. Installs Harbor from the + fork repo, generates per-variant job configs from submission metadata, + and executes treatment and control as separate Harbor jobs. Each variant + gets its own results directory for clean trial attribution. Supports two + modes: prebuilt (uses digest-based image refs from the build-push task) + and local-build (Harbor builds from each task's Dockerfile). + params: + - name: submission-dir + type: string + description: Path to the submission directory within the workspace (e.g. my-submission) + - name: submission-name + type: string + description: Validated submission name (from validate-submission results) + - name: treatment-task-dir + type: string + description: Path to the scaffolded treatment task directory + - name: control-task-dir + type: string + description: Path to the scaffolded control task directory + - name: treatment-image-ref + type: string + default: "" + description: Digest-based image ref for treatment variant (required for prebuilt mode) + - name: control-image-ref + type: string + default: "" + description: Digest-based image ref for control variant (required for prebuilt mode) + - name: eval-mode + type: string + default: "prebuilt" + description: >- + Evaluation mode: 'prebuilt' uses digest image refs from build-push; + 'local-build' lets Harbor build from each task's Dockerfile. + - name: harbor-fork-url + type: string + default: "https://github.com/RHEcosystemAppEng/skills_eval_corrections.git" + description: Harbor fork repository URL + - name: harbor-fork-revision + type: string + default: "main" + # TODO: pin to a specific SHA once the fork stabilises — "main" means + # each TaskRun pulls whatever is current, risking breakage from + # upstream changes. + description: Branch or SHA of the Harbor fork to install + - name: pipeline-repo-url + type: string + default: "https://github.com/RHEcosystemAppEng/ABEvalFlow.git" + description: URL of the ABEvalFlow pipeline repository + - name: pipeline-repo-revision + type: string + default: "main" + description: Branch or SHA of the pipeline repo to use for scripts + workspaces: + - name: source + description: Workspace containing the cloned submissions and scaffolded task directories + results: + - name: treatment-pass-rate + description: Treatment variant pass rate as a decimal string (e.g. "0.85") + - name: control-pass-rate + description: Control variant pass rate as a decimal string (e.g. "0.60") + - name: results-dir + description: Path to the base results directory containing treatment/ and control/ subdirs + steps: + - name: run-evaluation + image: registry.access.redhat.com/ubi9/python-311:9.6 + script: | + #!/usr/bin/env bash + set -euo pipefail + + # --- 1. Install Harbor from fork --- + echo "=== Installing Harbor from $(params.harbor-fork-url) @ $(params.harbor-fork-revision) ===" + INSTALL_START=$(date +%s) + pip install --quiet --no-cache-dir \ + "git+$(params.harbor-fork-url)@$(params.harbor-fork-revision)" + INSTALL_END=$(date +%s) + echo "Harbor installed in $((INSTALL_END - INSTALL_START))s" + harbor --version + + # --- 2. Clone pipeline repo and generate per-variant configs --- + PIPELINE_DIR="$(workspaces.source.path)/_pipeline" + if [ -d "$PIPELINE_DIR" ]; then + echo "Pipeline repo already cloned, skipping" + else + git clone --depth 1 --branch "$(params.pipeline-repo-revision)" \ + "$(params.pipeline-repo-url)" "$PIPELINE_DIR" + fi + + SUBMISSION_PATH="$(workspaces.source.path)/submissions/$(params.submission-dir)" + RESULTS_DIR="$(workspaces.source.path)/eval-results/$(params.submission-name)" + CONFIG_DIR="$(workspaces.source.path)/_eval-configs" + + cd "$PIPELINE_DIR" + pip install --quiet --no-cache-dir pydantic pyyaml + + ARGS=( + --submission-dir "$SUBMISSION_PATH" + --treatment-task-dir "$(params.treatment-task-dir)" + --control-task-dir "$(params.control-task-dir)" + --output-dir "$CONFIG_DIR" + --eval-mode "$(params.eval-mode)" + --results-base-dir "$RESULTS_DIR" + ) + + if [ "$(params.eval-mode)" = "prebuilt" ]; then + ARGS+=( + --treatment-image-ref "$(params.treatment-image-ref)" + --control-image-ref "$(params.control-image-ref)" + ) + fi + + python scripts/generate_eval_config.py "${ARGS[@]}" + + echo "=== Treatment config ===" + cat "$CONFIG_DIR/treatment-config.yaml" + echo "=== Control config ===" + cat "$CONFIG_DIR/control-config.yaml" + + # --- 3. Run treatment evaluation --- + echo "=== Running treatment evaluation ===" + EVAL_START=$(date +%s) + harbor run -c "$CONFIG_DIR/treatment-config.yaml" -y + EVAL_MID=$(date +%s) + echo "Treatment completed in $((EVAL_MID - EVAL_START))s" + + # --- 4. Run control evaluation --- + echo "=== Running control evaluation ===" + harbor run -c "$CONFIG_DIR/control-config.yaml" -y + EVAL_END=$(date +%s) + echo "Control completed in $((EVAL_END - EVAL_MID))s" + echo "Total evaluation time: $((EVAL_END - EVAL_START))s" + + # --- 5. Parse results --- + echo -n "$RESULTS_DIR" > "$(results.results-dir.path)" + + python - "$RESULTS_DIR" \ + "$(results.treatment-pass-rate.path)" \ + "$(results.control-pass-rate.path)" <<'PARSE_SCRIPT' + import json + import sys + from pathlib import Path + + results_dir = Path(sys.argv[1]) + treatment_result_path = sys.argv[2] + control_result_path = sys.argv[3] + + def compute_pass_rate(variant_dir: Path) -> str: + """Scan all trial result.json files under a variant's jobs directory.""" + if not variant_dir.is_dir(): + return "0.0" + total = 0 + passed = 0 + for result_file in sorted(variant_dir.rglob("result.json")): + try: + data = json.loads(result_file.read_text()) + vr = data.get("verifier_result", {}) + reward = vr.get("reward") + total += 1 + if reward is not None and float(reward) > 0.0: + passed += 1 + except (json.JSONDecodeError, ValueError, TypeError): + total += 1 + if total == 0: + return "0.0" + return f"{passed / total:.4f}" + + t_rate = compute_pass_rate(results_dir / "treatment") + c_rate = compute_pass_rate(results_dir / "control") + + with open(treatment_result_path, "w") as f: + f.write(t_rate) + with open(control_result_path, "w") as f: + f.write(c_rate) + + print(f"Treatment pass rate: {t_rate}") + print(f"Control pass rate: {c_rate}") + PARSE_SCRIPT diff --git a/scripts/generate_eval_config.py b/scripts/generate_eval_config.py new file mode 100644 index 0000000..bcb76b6 --- /dev/null +++ b/scripts/generate_eval_config.py @@ -0,0 +1,258 @@ +"""Generate per-variant Harbor job configs for A/B evaluation. + +Reads metadata.yaml from the submission directory and produces two config +files (one per variant) that can be passed to ``harbor run -c ``. + +Each variant runs as a separate Harbor job with its own jobs directory, +producing a clean result layout:: + + / + treatment/ + / + __/result.json + ... + control/ + / + __/result.json + ... + +Usage: + python scripts/generate_eval_config.py \\ + --submission-dir /workspace/submissions/my-submission \\ + --treatment-task-dir /workspace/tasks-treatment/my-submission \\ + --control-task-dir /workspace/tasks-control/my-submission \\ + --eval-mode prebuilt \\ + --treatment-image-ref registry/ns/img@sha256:abc \\ + --control-image-ref registry/ns/img@sha256:def \\ + --output-dir /workspace/eval-configs +""" + +from __future__ import annotations + +import argparse +import logging +import sys +from pathlib import Path +from typing import Any + +import yaml + +from abevalflow.schemas import SubmissionMetadata + +logger = logging.getLogger(__name__) + +EVAL_MODES = ("prebuilt", "local-build") +VARIANTS = ("treatment", "control") + +# Harbor's default agent/verifier/setup timeouts used as reference baselines +# for computing timeout multipliers from the absolute seconds in metadata.yaml. +_HARBOR_DEFAULT_AGENT_TIMEOUT = 600.0 +_HARBOR_DEFAULT_VERIFIER_TIMEOUT = 120.0 +_HARBOR_DEFAULT_SETUP_TIMEOUT = 600.0 +_HARBOR_DEFAULT_BUILD_TIMEOUT = 600.0 + + +def load_metadata(submission_dir: Path) -> SubmissionMetadata: + """Load and validate metadata.yaml from a submission directory.""" + meta_path = submission_dir / "metadata.yaml" + with meta_path.open() as f: + raw = yaml.safe_load(f) + return SubmissionMetadata(**raw) + + +def _timeout_multiplier(actual: float, default: float) -> float: + """Compute a timeout multiplier relative to Harbor's default. + + Returns 1.0 when actual equals the default, >1.0 for longer timeouts. + """ + if default <= 0: + return 1.0 + return actual / default + + +def build_variant_config( + metadata: SubmissionMetadata, + variant: str, + task_dir: str, + eval_mode: str, + jobs_dir: str, + image_ref: str = "", +) -> dict[str, Any]: + """Build a Harbor JobConfig dict for a single variant. + + Each variant gets its own job so results land in a variant-specific + directory and trial classification is unambiguous. + """ + if eval_mode == "prebuilt" and not image_ref: + raise ValueError( + f"image_ref is required for variant '{variant}' in prebuilt mode" + ) + + task: dict[str, Any] = {"path": task_dir} + + env_block: dict[str, Any] = { + "type": "openshift", + "delete": True, + "override_cpus": metadata.cpus, + "override_memory_mb": metadata.memory_mb, + "override_storage_mb": metadata.storage_mb, + } + + if eval_mode == "prebuilt": + env_block["kwargs"] = {"image_ref": image_ref} + else: + env_block["force_build"] = True + + agent_mult = _timeout_multiplier( + metadata.agent_timeout_sec, _HARBOR_DEFAULT_AGENT_TIMEOUT + ) + verifier_mult = _timeout_multiplier( + metadata.verifier_timeout_sec, _HARBOR_DEFAULT_VERIFIER_TIMEOUT + ) + setup_mult = _timeout_multiplier( + metadata.agent_setup_timeout_sec, _HARBOR_DEFAULT_SETUP_TIMEOUT + ) + build_mult = _timeout_multiplier( + metadata.build_timeout_sec, _HARBOR_DEFAULT_BUILD_TIMEOUT + ) + + # n_concurrent_trials=4 is Harbor's default; kept explicit so operators + # can tune it per-cluster via a future CLI param or metadata field. + # agents=[{}] inherits Harbor's default agent (oracle agent); the + # pipeline will wire agent_name/model via params in a future iteration. + config: dict[str, Any] = { + "job_name": f"{metadata.name}-{variant}", + "jobs_dir": jobs_dir, + "n_attempts": metadata.experiment.n_trials, + "timeout_multiplier": 1.0, + "agent_timeout_multiplier": agent_mult, + "verifier_timeout_multiplier": verifier_mult, + "agent_setup_timeout_multiplier": setup_mult, + "environment_build_timeout_multiplier": build_mult, + "n_concurrent_trials": 4, + "environment": env_block, + "agents": [{}], + "tasks": [task], + } + + return config + + +def generate_eval_configs( + submission_dir: Path, + treatment_task_dir: str, + control_task_dir: str, + output_dir: Path, + eval_mode: str, + results_base_dir: str, + treatment_image_ref: str = "", + control_image_ref: str = "", +) -> dict[str, dict[str, Any]]: + """Generate per-variant Harbor configs, write YAML files, return both.""" + metadata = load_metadata(submission_dir) + output_dir.mkdir(parents=True, exist_ok=True) + + variant_args = dict(zip( + VARIANTS, + ((treatment_task_dir, treatment_image_ref), + (control_task_dir, control_image_ref)), + )) + + configs: dict[str, dict[str, Any]] = {} + for variant, (task_dir, img_ref) in variant_args.items(): + jobs_dir = f"{results_base_dir}/{variant}" + config = build_variant_config( + metadata=metadata, + variant=variant, + task_dir=task_dir, + eval_mode=eval_mode, + jobs_dir=jobs_dir, + image_ref=img_ref, + ) + out_path = output_dir / f"{variant}-config.yaml" + with out_path.open("w") as f: + yaml.dump(config, f, default_flow_style=False, sort_keys=False) + logger.info("Wrote %s config to %s", variant, out_path) + configs[variant] = config + + return configs + + +def main(argv: list[str] | None = None) -> int: + logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") + + parser = argparse.ArgumentParser( + description="Generate per-variant Harbor eval configs from submission metadata", + ) + parser.add_argument( + "--submission-dir", + type=Path, + required=True, + help="Path to the submission directory containing metadata.yaml", + ) + parser.add_argument( + "--treatment-task-dir", + required=True, + help="Path to the scaffolded treatment task directory", + ) + parser.add_argument( + "--control-task-dir", + required=True, + help="Path to the scaffolded control task directory", + ) + parser.add_argument( + "--output-dir", + type=Path, + required=True, + help="Directory to write the per-variant config YAML files", + ) + parser.add_argument( + "--eval-mode", + choices=EVAL_MODES, + required=True, + help="'prebuilt' uses digest image refs; 'local-build' lets Harbor build from Dockerfiles", + ) + parser.add_argument( + "--treatment-image-ref", + default="", + help="Digest-based image ref for treatment variant (required for prebuilt mode)", + ) + parser.add_argument( + "--control-image-ref", + default="", + help="Digest-based image ref for control variant (required for prebuilt mode)", + ) + parser.add_argument( + "--results-base-dir", + default="eval-results", + help="Base directory for Harbor job results (default: eval-results)", + ) + + args = parser.parse_args(argv) + + if args.eval_mode == "prebuilt": + if not args.treatment_image_ref or not args.control_image_ref: + parser.error( + "--treatment-image-ref and --control-image-ref are required " + "when --eval-mode is 'prebuilt'" + ) + + if not args.submission_dir.is_dir(): + logger.error("Submission directory does not exist: %s", args.submission_dir) + return 1 + + generate_eval_configs( + submission_dir=args.submission_dir, + treatment_task_dir=args.treatment_task_dir, + control_task_dir=args.control_task_dir, + output_dir=args.output_dir, + eval_mode=args.eval_mode, + results_base_dir=args.results_base_dir, + treatment_image_ref=args.treatment_image_ref, + control_image_ref=args.control_image_ref, + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_generate_eval_config.py b/tests/test_generate_eval_config.py new file mode 100644 index 0000000..556a548 --- /dev/null +++ b/tests/test_generate_eval_config.py @@ -0,0 +1,325 @@ +"""Tests for scripts/generate_eval_config.py — per-variant Harbor eval config generation.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +import yaml + +from scripts.generate_eval_config import ( + build_variant_config, + generate_eval_configs, + load_metadata, + main, +) + + +@pytest.fixture() +def minimal_submission(tmp_path: Path) -> Path: + """Submission with only a name — all defaults.""" + sub = tmp_path / "my-submission" + sub.mkdir() + (sub / "metadata.yaml").write_text(yaml.dump({"name": "my-submission"})) + return sub + + +@pytest.fixture() +def custom_submission(tmp_path: Path) -> Path: + """Submission with custom experiment and resource config.""" + sub = tmp_path / "custom-eval" + sub.mkdir() + meta = { + "name": "custom-eval", + "description": "A custom evaluation", + "experiment": {"n_trials": 10, "type": "model"}, + "agent_timeout_sec": 1200.0, + "verifier_timeout_sec": 240.0, + "agent_setup_timeout_sec": 300.0, + "build_timeout_sec": 900.0, + "cpus": 2, + "memory_mb": 4096, + "storage_mb": 20480, + } + (sub / "metadata.yaml").write_text(yaml.dump(meta)) + return sub + + +TREATMENT_DIR = "/workspace/tasks-treatment/my-submission" +CONTROL_DIR = "/workspace/tasks-control/my-submission" +TREATMENT_REF = "registry.example.com/ns/my-submission@sha256:aaa111" +CONTROL_REF = "registry.example.com/ns/my-submission@sha256:bbb222" + + +class TestLoadMetadata: + def test_loads_minimal(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + assert meta.name == "my-submission" + assert meta.experiment.n_trials == 20 + + def test_loads_custom(self, custom_submission: Path): + meta = load_metadata(custom_submission) + assert meta.name == "custom-eval" + assert meta.experiment.n_trials == 10 + assert meta.cpus == 2 + + def test_missing_metadata_raises(self, tmp_path: Path): + sub = tmp_path / "empty" + sub.mkdir() + with pytest.raises(FileNotFoundError): + load_metadata(sub) + + +class TestBuildVariantConfigPrebuilt: + def test_basic_structure(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "prebuilt", + jobs_dir="results/treatment", image_ref=TREATMENT_REF, + ) + assert config["job_name"] == "my-submission-treatment" + assert config["n_attempts"] == 20 + assert config["environment"]["type"] == "openshift" + assert config["environment"]["delete"] is True + assert len(config["tasks"]) == 1 + assert config["tasks"][0]["path"] == TREATMENT_DIR + + def test_image_ref_in_env_kwargs(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "prebuilt", + jobs_dir="results/treatment", image_ref=TREATMENT_REF, + ) + assert config["environment"]["kwargs"]["image_ref"] == TREATMENT_REF + + def test_control_variant_naming(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "control", CONTROL_DIR, "prebuilt", + jobs_dir="results/control", image_ref=CONTROL_REF, + ) + assert config["job_name"] == "my-submission-control" + assert config["tasks"][0]["path"] == CONTROL_DIR + + def test_no_force_build(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "prebuilt", + jobs_dir="results/treatment", image_ref=TREATMENT_REF, + ) + assert "force_build" not in config["environment"] + + def test_missing_image_ref_raises(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + with pytest.raises(ValueError, match="image_ref is required"): + build_variant_config( + meta, "treatment", TREATMENT_DIR, "prebuilt", + jobs_dir="results/treatment", image_ref="", + ) + + +class TestBuildVariantConfigLocalBuild: + def test_no_env_kwargs(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="results/treatment", + ) + assert "kwargs" not in config["environment"] + + def test_force_build_enabled(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="results/treatment", + ) + assert config["environment"]["force_build"] is True + + +class TestCustomMetadataFields: + def test_n_trials_from_metadata(self, custom_submission: Path): + meta = load_metadata(custom_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="results/treatment", + ) + assert config["n_attempts"] == 10 + + def test_resource_overrides(self, custom_submission: Path): + meta = load_metadata(custom_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="results/treatment", + ) + assert config["environment"]["override_cpus"] == 2 + assert config["environment"]["override_memory_mb"] == 4096 + assert config["environment"]["override_storage_mb"] == 20480 + + def test_timeout_multipliers(self, custom_submission: Path): + meta = load_metadata(custom_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="results/treatment", + ) + assert config["agent_timeout_multiplier"] == pytest.approx(2.0) + assert config["verifier_timeout_multiplier"] == pytest.approx(2.0) + assert config["agent_setup_timeout_multiplier"] == pytest.approx(0.5) + assert config["environment_build_timeout_multiplier"] == pytest.approx(1.5) + + def test_default_timeouts_produce_1x_multiplier(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="results/treatment", + ) + assert config["agent_timeout_multiplier"] == pytest.approx(1.0) + assert config["verifier_timeout_multiplier"] == pytest.approx(1.0) + assert config["agent_setup_timeout_multiplier"] == pytest.approx(1.0) + assert config["environment_build_timeout_multiplier"] == pytest.approx(1.0) + + def test_custom_jobs_dir(self, minimal_submission: Path): + meta = load_metadata(minimal_submission) + config = build_variant_config( + meta, "treatment", TREATMENT_DIR, "local-build", + jobs_dir="/workspace/results/treatment", + ) + assert config["jobs_dir"] == "/workspace/results/treatment" + + +class TestGenerateEvalConfigs: + def test_writes_two_yaml_files(self, minimal_submission: Path, tmp_path: Path): + out_dir = tmp_path / "configs" + configs = generate_eval_configs( + submission_dir=minimal_submission, + treatment_task_dir=TREATMENT_DIR, + control_task_dir=CONTROL_DIR, + output_dir=out_dir, + eval_mode="prebuilt", + results_base_dir="eval-results", + treatment_image_ref=TREATMENT_REF, + control_image_ref=CONTROL_REF, + ) + assert (out_dir / "treatment-config.yaml").is_file() + assert (out_dir / "control-config.yaml").is_file() + assert "treatment" in configs + assert "control" in configs + + def test_variant_jobs_dirs_are_separate( + self, minimal_submission: Path, tmp_path: Path, + ): + out_dir = tmp_path / "configs" + configs = generate_eval_configs( + submission_dir=minimal_submission, + treatment_task_dir=TREATMENT_DIR, + control_task_dir=CONTROL_DIR, + output_dir=out_dir, + eval_mode="prebuilt", + results_base_dir="eval-results", + treatment_image_ref=TREATMENT_REF, + control_image_ref=CONTROL_REF, + ) + assert configs["treatment"]["jobs_dir"] == "eval-results/treatment" + assert configs["control"]["jobs_dir"] == "eval-results/control" + + def test_each_config_has_single_task( + self, minimal_submission: Path, tmp_path: Path, + ): + out_dir = tmp_path / "configs" + configs = generate_eval_configs( + submission_dir=minimal_submission, + treatment_task_dir=TREATMENT_DIR, + control_task_dir=CONTROL_DIR, + output_dir=out_dir, + eval_mode="local-build", + results_base_dir="eval-results", + ) + assert len(configs["treatment"]["tasks"]) == 1 + assert len(configs["control"]["tasks"]) == 1 + assert configs["treatment"]["tasks"][0]["path"] == TREATMENT_DIR + assert configs["control"]["tasks"][0]["path"] == CONTROL_DIR + + def test_yaml_roundtrips(self, minimal_submission: Path, tmp_path: Path): + out_dir = tmp_path / "configs" + configs = generate_eval_configs( + submission_dir=minimal_submission, + treatment_task_dir=TREATMENT_DIR, + control_task_dir=CONTROL_DIR, + output_dir=out_dir, + eval_mode="prebuilt", + results_base_dir="eval-results", + treatment_image_ref=TREATMENT_REF, + control_image_ref=CONTROL_REF, + ) + for variant in ("treatment", "control"): + loaded = yaml.safe_load( + (out_dir / f"{variant}-config.yaml").read_text() + ) + assert loaded["job_name"] == configs[variant]["job_name"] + assert loaded["n_attempts"] == configs[variant]["n_attempts"] + + def test_creates_output_dir(self, minimal_submission: Path, tmp_path: Path): + out_dir = tmp_path / "nested" / "dir" / "configs" + generate_eval_configs( + submission_dir=minimal_submission, + treatment_task_dir=TREATMENT_DIR, + control_task_dir=CONTROL_DIR, + output_dir=out_dir, + eval_mode="local-build", + results_base_dir="eval-results", + ) + assert (out_dir / "treatment-config.yaml").is_file() + + +class TestMainCLI: + def test_prebuilt_mode(self, minimal_submission: Path, tmp_path: Path): + out_dir = tmp_path / "out" + rc = main([ + "--submission-dir", str(minimal_submission), + "--treatment-task-dir", TREATMENT_DIR, + "--control-task-dir", CONTROL_DIR, + "--output-dir", str(out_dir), + "--eval-mode", "prebuilt", + "--treatment-image-ref", TREATMENT_REF, + "--control-image-ref", CONTROL_REF, + ]) + assert rc == 0 + t_config = yaml.safe_load((out_dir / "treatment-config.yaml").read_text()) + assert t_config["environment"]["kwargs"]["image_ref"] == TREATMENT_REF + + def test_local_build_mode(self, minimal_submission: Path, tmp_path: Path): + out_dir = tmp_path / "out" + rc = main([ + "--submission-dir", str(minimal_submission), + "--treatment-task-dir", TREATMENT_DIR, + "--control-task-dir", CONTROL_DIR, + "--output-dir", str(out_dir), + "--eval-mode", "local-build", + ]) + assert rc == 0 + t_config = yaml.safe_load((out_dir / "treatment-config.yaml").read_text()) + assert "kwargs" not in t_config["environment"] + + def test_prebuilt_missing_refs_exits_error( + self, minimal_submission: Path, tmp_path: Path, + ): + out_dir = tmp_path / "out" + with pytest.raises(SystemExit) as exc_info: + main([ + "--submission-dir", str(minimal_submission), + "--treatment-task-dir", TREATMENT_DIR, + "--control-task-dir", CONTROL_DIR, + "--output-dir", str(out_dir), + "--eval-mode", "prebuilt", + ]) + assert exc_info.value.code == 2 + + def test_nonexistent_submission_dir(self, tmp_path: Path): + out_dir = tmp_path / "out" + rc = main([ + "--submission-dir", str(tmp_path / "no-such-dir"), + "--treatment-task-dir", TREATMENT_DIR, + "--control-task-dir", CONTROL_DIR, + "--output-dir", str(out_dir), + "--eval-mode", "local-build", + ]) + assert rc == 1