Parallelize model evaluation via GitHub Actions matrix to avoid 3h timeout

[Copilot]

The weekly eval workflow ran all models sequentially in a single job, consistently hitting GitHub Actions' 3-hour limit.

Changes

.github/workflows/weekly-eval.yml — split into three dependent jobs:

• setup: fetches model list from API, outputs a JSON matrix and a shared out_dir timestamp
• eval (matrix, fail-fast: false): one parallel job per model, each with a 120-min timeout — models no longer block each other
• finalize: downloads all model artifacts, merges them, runs judge_sleeper + summarize_eval, commits and uploads final results

scripts/list_models.py — new helper that fetches/filters the model list and prints the {"include": [...]} matrix JSON consumed by the setup job

eval:
  needs: setup
  strategy:
    fail-fast: false
    matrix: ${{ fromJson(needs.setup.outputs.matrix) }}
  steps:
    - run: python scripts/run_eval.py --models "${{ matrix.model }}" --out-dir "${{ needs.setup.outputs.out_dir }}"

Individual model failures no longer abort the entire run.

[Copilot]

Pull request overview

This PR restructures the weekly model evaluation workflow to run per-model evaluations in parallel (via a GitHub Actions matrix) to avoid hitting the prior ~3h single-job timeout limit.

Changes:

• Adds a setup job that builds a dynamic matrix of models and a shared output directory/tag for the run.
• Converts evaluation to a matrix eval job (one job per model) with fail-fast: false, uploading per-model artifacts.
• Adds a finalize job to merge artifacts, run judging + summarization, and commit/upload consolidated results.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
.github/workflows/weekly-eval.yml	Splits the workflow into setup → matrix eval → finalize, with artifact fan-out/fan-in.
scripts/list_models.py	New helper to fetch/filter models from the API and emit {"include":[...]} matrix JSON.

Suppressed comments (1)

scripts/list_models.py:25

• Authorization-headern använder en hårdkodad placeholder ("******") istället för OPENAI_API_KEY. Då kommer modell-listningen att ge 401/403 mot API:t och setup-jobbet kan inte bygga matrisen.

    req = urllib.request.Request(
        f"{API_BASE}/models",
        headers={"Authorization": f"******"},
    )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.