From 7a54b9867c7697319136bb8249956e9c2c86a6a4 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 10:00:49 +0200 Subject: [PATCH 01/18] docs(specs): add eval PR fork dispatch design Design for secure eval execution on fork PRs using workflow_run dispatch with collaborator-based trust gating. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...2026-05-07-eval-pr-fork-dispatch-design.md | 295 ++++++++++++++++++ 1 file changed, 295 insertions(+) create mode 100644 docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md diff --git a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md new file mode 100644 index 000000000..ceab41c94 --- /dev/null +++ b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md @@ -0,0 +1,295 @@ +# Eval PR Fork Dispatch + +**Date**: 2026-05-07 +**Scope**: Secure eval execution for fork PRs via `workflow_run` dispatch +**Relates to**: [Eval Skills CI Workflow](2026-04-21-eval-skills-ci-workflow-design.md) + +## Problem + +The `eval-pr.yml` workflow requires repository secrets (`GCP_SA_KEY`, `CLOUD_ML_REGION`) to authenticate to Google Cloud for running Claude Code via Vertex AI. GitHub does not pass secrets to workflows triggered from forked repositories: + +> "With the exception of `GITHUB_TOKEN`, secrets are not passed to the runner when a workflow is triggered from a forked repository." +> — [Using secrets in GitHub Actions](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions) + +This means fork PRs that modify skills or evals cannot produce eval results. + +## Solution + +Split `eval-pr.yml` into two workflows using the `workflow_run` event, which runs in the base repository context with full secret access: + +> "The workflow started by the `workflow_run` event is able to access secrets and write tokens, even if the previous workflow was not." +> — [Events that trigger workflows: workflow_run](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#workflow_run) + +A trust check auto-approves eval runs for repository collaborators with write access, while external contributors require manual approval via a GitHub protected environment. + +## Design Decisions + +| Decision | Choice | Rationale | +|----------|--------|-----------| +| Dispatch mechanism | `workflow_run` | Only mechanism that works for fork PRs — `workflow_dispatch`/`repository_dispatch` require write tokens that fork PR `GITHUB_TOKEN` lacks | +| Trust signal | Collaborator permission API | `GET /repos/{owner}/{repo}/collaborators/{username}/permission` — uses GitHub's actual permission model, handles teams automatically, no file parsing | +| Trust threshold | `admin` or `write` | Collaborators with write access can already push to the repo directly — granting them secret access in CI adds no new attack surface | +| Untrusted gate | Protected environment | GitHub's built-in approval mechanism — reviewers click "Approve" in the Actions UI | +| PR coverage | All PRs (fork and non-fork) | Single code path avoids duplicate logic and ensures consistent behavior | +| Artifact naming | `eval-pr-metadata` (fixed name) | Artifacts are scoped to workflow runs — concurrent PRs each get their own run ID, no collision | + +## Architecture + +``` +Fork PR opened + │ + ▼ +┌──────────────────────────────┐ +│ eval-pr.yml (pull_request) │ No secrets needed +│ 1. Checkout │ +│ 2. Discover changed skills │ +│ 3. Upload artifact │ +│ (skills, PR#, SHA, author)│ +└──────────────┬───────────────┘ + │ workflow_run (completed) + ▼ +┌──────────────────────────────────────┐ +│ eval-pr-run.yml (workflow_run) │ Base repo context — secrets available +│ │ +│ Job 1: discover │ +│ - Download artifact │ +│ - Check collaborator permission │ +│ - Output: trusted, skills, PR#, SHA │ +│ │ +│ Job 2: gate │ +│ - Runs ONLY if trusted != true │ +│ - environment: eval-protected │ +│ - Blocks until reviewer approves │ +│ │ +│ Job 3: run-evals │ +│ - Checkout PR merge commit │ +│ - Authenticate GCP │ +│ - Run evals │ +│ - Post PR review │ +└──────────────────────────────────────┘ +``` + +## Workflow 1: eval-pr.yml (modified) + +### Changes from Current + +The workflow is stripped to discovery and artifact upload only. All secret-dependent steps are removed: GCP authentication, Claude Code installation, eval execution, and review posting. + +### Trigger + +Unchanged: + +```yaml +on: + pull_request: + branches: [main] + paths: + - 'plugins/sdlc-workflow/skills/**/*.md' + - 'evals/**/evals.json' + - '.github/workflows/eval-pr.yml' +``` + +### Permissions + +```yaml +permissions: + contents: read +``` + +Reduced from `contents: read` + `pull-requests: write` — this workflow no longer posts reviews. + +### Steps + +1. **Checkout** — `actions/checkout@v4` with `fetch-depth: 0` +2. **Discover changed skills** — existing logic, unchanged (git diff against PR base, map to eval suites) +3. **Write metadata file** — write `eval-pr-metadata.json` with PR context: + +```json +{ + "skills": "skill1,skill2", + "pr_number": 90, + "head_sha": "abc123def", + "author": "username" +} +``` + +4. **Upload artifact** — `actions/upload-artifact@v4` with name `eval-pr-metadata`, containing the `eval-pr-metadata.json` file. + +The artifact is uploaded even when `skills` is empty so Stage 2 can detect the no-op cleanly. + +## Workflow 2: eval-pr-run.yml (new) + +### Trigger + +```yaml +on: + workflow_run: + workflows: ["Eval PR"] + types: [completed] +``` + +No path filtering — `workflow_run` does not support it. The `completed` type fires on both success and failure, so the `discover` job checks `github.event.workflow_run.conclusion == 'success'` before proceeding. If the triggering workflow found no changed skills, the `run-evals` job skips via its `if` condition. + +### Permissions + +```yaml +permissions: + contents: read + pull-requests: write + actions: read +``` + +`actions: read` is needed to download artifacts from the triggering run. + +### Job 1: discover + +Downloads the artifact and determines trust level. Skips early if the triggering workflow failed. + +**Steps:** + +1. **Check triggering workflow conclusion** — exit early if `github.event.workflow_run.conclusion != 'success'` + +2. **Download artifact** from triggering run: + +```yaml +- uses: actions/download-artifact@v4 + with: + name: eval-pr-metadata + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} +``` + +3. **Parse metadata and check trust** via `actions/github-script@v7`: + +```javascript +const metadata = JSON.parse(fs.readFileSync('eval-pr-metadata.json', 'utf8')); + +let trusted = false; +try { + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username: metadata.author + }); + trusted = ['admin', 'write'].includes(data.permission); +} catch (e) { + // Non-collaborators may 404 — treat as untrusted + trusted = false; +} +``` + +4. **Set outputs**: `skills`, `pr_number`, `head_sha`, `author`, `trusted` + +### Job 2: gate + +```yaml +gate: + needs: discover + if: needs.discover.outputs.trusted != 'true' && needs.discover.outputs.skills != '' + runs-on: ubuntu-latest + environment: eval-protected + steps: + - run: echo "Approved by reviewer" +``` + +Skipped entirely for trusted authors and when there are no skills to evaluate. For untrusted authors, the job is created but paused until a designated reviewer approves it in the Actions UI. + +### Job 3: run-evals + +```yaml +run-evals: + needs: [discover, gate] + if: >- + !cancelled() && + needs.discover.outputs.skills != '' && + (needs.discover.outputs.trusted == 'true' || needs.gate.result == 'success') + runs-on: ubuntu-latest +``` + +**Steps:** + +1. **Checkout PR merge commit**: + +```yaml +- uses: actions/checkout@v4 + with: + ref: refs/pull/${{ needs.discover.outputs.pr_number }}/merge + fetch-depth: 0 +``` + +2. **Authenticate to Google Cloud** — existing step, moved from `eval-pr.yml`: + +```yaml +- uses: google-github-actions/auth@v3 + with: + credentials_json: ${{ secrets.GCP_SA_KEY }} +``` + +3. **Install Claude Code** — existing step, moved from `eval-pr.yml` + +4. **Run PR evals** — existing step, moved from `eval-pr.yml`. Uses `needs.discover.outputs.skills` instead of the inline discovery output. + +5. **Post eval results review** — existing step, moved from `eval-pr.yml`. Uses `needs.discover.outputs.pr_number` instead of `context.issue.number` (which is unavailable in `workflow_run` context). + +## Trust Model + +### Trusted Authors (auto-approve) + +Repository collaborators with `write` or `admin` permission. Determined by the GitHub REST API: + +``` +GET /repos/{owner}/{repo}/collaborators/{username}/permission +``` + +This returns the highest permission level across all grant sources (direct, teams, org, enterprise). Collaborators with write access can already push directly to the repository, so granting them secret access in CI adds no new attack surface. + +### Untrusted Authors (manual approval) + +All other PR authors. Their eval runs require a reviewer to click "Approve" in the GitHub Actions UI. The protected environment `eval-protected` enforces this gate. + +### Security Properties + +- The `workflow_run` workflow executes on the base repository's default branch — a fork PR cannot modify the workflow definition or trust logic +- The collaborator permission check uses GitHub's API, not a file in the repo — it cannot be manipulated by a PR +- Once approved, the workflow does execute skill files from the PR with access to secrets via environment variables — this is inherent to the use case (evaluating skills requires running them) +- The `dontAsk` permission mode with explicit `--allowedTools` limits Claude Code's capabilities, but a malicious skill could still read environment variables via the Bash tool + +### Why Not Other Approaches + +| Approach | Why not | +|----------|---------| +| `workflow_dispatch` / `repository_dispatch` | Cannot be triggered from fork PR workflows — `GITHUB_TOKEN` is read-only for fork PRs | +| `pull_request_target` | Runs in base repo context with secrets, but GitHub warns: "Avoid using this event if you need to build or run code from the pull request" | +| CODEOWNERS parsing | Text file parsing is fragile (teams, globs, multiple entries); collaborator API is authoritative | +| Skip evals for fork PRs | Fork contributors would never see eval results | + +## Setup Requirements + +### GitHub Environment + +Create a protected environment in the repository: + +1. Settings > Environments > New environment: `eval-protected` +2. Add required reviewers (at minimum, repository maintainers) +3. No environment-specific secrets needed — existing repository-level secrets (`GCP_SA_KEY`, `CLOUD_ML_REGION`) are available to all `workflow_run` jobs + +### No Other Infrastructure + +- No CODEOWNERS file needed +- No GitHub App or PAT required +- No changes to `eval-baseline.yml` (runs on push to main, secrets always available) + +## File Changes + +| File | Change | +|------|--------| +| `.github/workflows/eval-pr.yml` | Strip to discovery + artifact upload only | +| `.github/workflows/eval-pr-run.yml` | New workflow: trust check, gate, eval execution, review posting | +| `docs/specs/2026-04-21-eval-skills-ci-workflow-design.md` | Add reference to this spec | + +## References + +- [Using secrets in GitHub Actions](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions) — fork PR secret restriction +- [Events that trigger workflows: workflow_run](https://docs.github.com/en/actions/writing-workflows/choosing-when-your-workflow-runs/events-that-trigger-workflows#workflow_run) — secret access in `workflow_run` +- [GitHub Security Lab: Preventing pwn requests](https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/) — two-stage pattern for fork PR processing +- [Repository collaborators API](https://docs.github.com/en/rest/collaborators/collaborators) — permission check endpoint From 62e083af4b575417fea9e7befa3d1db65797e36a Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 10:08:21 +0200 Subject: [PATCH 02/18] docs(plans): add eval PR fork dispatch implementation plan Three tasks: modify eval-pr.yml to discovery-only, create eval-pr-run.yml dispatch workflow, update spec cross-reference. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../plans/2026-05-07-eval-pr-fork-dispatch.md | 371 ++++++++++++++++++ 1 file changed, 371 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md diff --git a/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md b/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md new file mode 100644 index 000000000..ec11cecd2 --- /dev/null +++ b/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md @@ -0,0 +1,371 @@ +# Eval PR Fork Dispatch Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Split `eval-pr.yml` into a two-stage `workflow_run` dispatch pattern so fork PRs can run evals with secret access, gated by a collaborator trust check. + +**Architecture:** Stage 1 (`eval-pr.yml`, `pull_request` trigger) discovers changed skills and uploads an artifact with PR metadata. Stage 2 (`eval-pr-run.yml`, `workflow_run` trigger) downloads the artifact, checks if the PR author is a trusted collaborator, gates untrusted authors behind a protected environment, then runs evals and posts the PR review. + +**Tech Stack:** GitHub Actions, `actions/upload-artifact@v4`, `actions/download-artifact@v4`, `actions/github-script@v7`, GitHub REST API (collaborator permissions) + +**Spec:** `docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md` + +--- + +### Task 1: Modify eval-pr.yml to discovery-only with artifact upload + +Strip all secret-dependent steps (GCP auth, Claude Code install, eval execution, review posting) and add metadata write + artifact upload. + +**Files:** +- Modify: `.github/workflows/eval-pr.yml` + +- [ ] **Step 1: Rewrite eval-pr.yml** + +Replace the entire file with the discovery-only version. The job is renamed from `eval-pr` to `discover` to reflect its new purpose. Permissions drop to `contents: read` only. + +```yaml +# Discovers changed skills for PR evals and uploads metadata for the +# dispatch workflow (eval-pr-run.yml) to execute with secret access. +# +# See docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md for full design. + +name: Eval PR + +on: + pull_request: + branches: [main] + paths: + - 'plugins/sdlc-workflow/skills/**/*.md' + - 'evals/**/evals.json' + - '.github/workflows/eval-pr.yml' + +jobs: + discover: + name: Discover Changed Skills + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Discover changed skills + id: discover + env: + BASE_SHA: ${{ github.event.pull_request.base.sha }} + run: | + changed_files=$(git diff --name-only "$BASE_SHA"...HEAD) + echo "Changed files:" + echo "$changed_files" + + skills="" + for file in $changed_files; do + skill="" + if [[ "$file" =~ ^plugins/sdlc-workflow/skills/([^/]+)/ ]]; then + skill="${BASH_REMATCH[1]}" + elif [[ "$file" =~ ^evals/([^/]+)/ ]]; then + skill="${BASH_REMATCH[1]}" + fi + + if [ -n "$skill" ] && [ -f "evals/${skill}/evals.json" ]; then + if [[ ! ",$skills," == *",$skill,"* ]]; then + skills="${skills:+${skills},}${skill}" + fi + fi + done + + echo "skills=${skills}" >> "$GITHUB_OUTPUT" + echo "Discovered changed skills with evals: ${skills:-none}" + + - name: Write PR metadata + env: + SKILLS: ${{ steps.discover.outputs.skills }} + run: | + jq -n \ + --arg skills "$SKILLS" \ + --argjson pr_number ${{ github.event.pull_request.number }} \ + --arg head_sha "${{ github.event.pull_request.head.sha }}" \ + --arg author "${{ github.event.pull_request.user.login }}" \ + '{skills: $skills, pr_number: $pr_number, head_sha: $head_sha, author: $author}' \ + > eval-pr-metadata.json + cat eval-pr-metadata.json + + - name: Upload PR metadata + uses: actions/upload-artifact@v4 + with: + name: eval-pr-metadata + path: eval-pr-metadata.json +``` + +- [ ] **Step 2: Validate YAML syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/eval-pr.yml')); print('Valid YAML')"` + +Expected: `Valid YAML` + +- [ ] **Step 3: Commit** + +```bash +git add .github/workflows/eval-pr.yml +git commit -m "refactor(ci): strip eval-pr.yml to discovery-only with artifact upload + +Stage 1 of the workflow_run dispatch pattern for fork PR secret access. +Eval execution moves to eval-pr-run.yml (next commit)." +``` + +--- + +### Task 2: Create eval-pr-run.yml dispatch workflow + +The new workflow with three jobs: `discover` (download artifact + trust check), `gate` (environment approval for untrusted authors), and `run-evals` (checkout PR, run evals, post review). + +**Files:** +- Create: `.github/workflows/eval-pr-run.yml` + +- [ ] **Step 1: Create eval-pr-run.yml** + +```yaml +# Runs evals for PRs via workflow_run dispatch, enabling secret access +# for fork PRs. Pairs with eval-pr.yml (Stage 1: discovery). +# +# Trust model: +# - Collaborators with write/admin permission: evals run automatically +# - External contributors: require manual approval via eval-protected environment +# +# See docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md for full design. + +name: Eval PR Run + +on: + workflow_run: + workflows: ["Eval PR"] + types: [completed] + +permissions: + contents: read + pull-requests: write + actions: read + +jobs: + discover: + name: Discover PR Context + if: github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest + outputs: + skills: ${{ steps.metadata.outputs.skills }} + pr_number: ${{ steps.metadata.outputs.pr_number }} + head_sha: ${{ steps.metadata.outputs.head_sha }} + author: ${{ steps.metadata.outputs.author }} + trusted: ${{ steps.metadata.outputs.trusted }} + steps: + - name: Download PR metadata + uses: actions/download-artifact@v4 + with: + name: eval-pr-metadata + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + + - name: Parse metadata and check trust + id: metadata + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const metadata = JSON.parse(fs.readFileSync('eval-pr-metadata.json', 'utf8')); + + core.setOutput('skills', metadata.skills); + core.setOutput('pr_number', metadata.pr_number.toString()); + core.setOutput('head_sha', metadata.head_sha); + core.setOutput('author', metadata.author); + + let trusted = false; + try { + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username: metadata.author + }); + trusted = ['admin', 'write'].includes(data.permission); + console.log(`Author ${metadata.author}: permission=${data.permission}, trusted=${trusted}`); + } catch (e) { + console.log(`Author ${metadata.author}: not a collaborator (${e.status}), trusted=false`); + } + + core.setOutput('trusted', trusted.toString()); + + gate: + name: Approval Gate + needs: discover + if: needs.discover.outputs.trusted != 'true' && needs.discover.outputs.skills != '' + runs-on: ubuntu-latest + environment: eval-protected + steps: + - run: echo "Approved by reviewer" + + run-evals: + name: Run PR Evals + needs: [discover, gate] + if: >- + !cancelled() && + needs.discover.result == 'success' && + needs.discover.outputs.skills != '' && + (needs.discover.outputs.trusted == 'true' || needs.gate.result == 'success') + runs-on: ubuntu-latest + steps: + - name: Checkout PR merge commit + uses: actions/checkout@v4 + with: + ref: refs/pull/${{ needs.discover.outputs.pr_number }}/merge + fetch-depth: 0 + + - name: Authenticate to Google Cloud + uses: google-github-actions/auth@v3 + with: + credentials_json: ${{ secrets.GCP_SA_KEY }} + + - name: Install Claude Code + run: curl -fsSL https://claude.ai/install.sh | bash + + - name: Run PR evals + env: + CLAUDE_CODE_USE_VERTEX: "1" + CLOUD_ML_REGION: ${{ secrets.GCP_CLOUD_ML_REGION }} + ANTHROPIC_DEFAULT_SONNET_MODEL: "claude-sonnet-4-6" + ANTHROPIC_DEFAULT_OPUS_MODEL: "claude-opus-4-6" + ANTHROPIC_MODEL: "claude-opus-4-6" + run: | + IFS=',' read -ra SKILLS <<< "${{ needs.discover.outputs.skills }}" + + for skill in "${SKILLS[@]}"; do + eval_count=$(jq '.evals | length' "evals/${skill}/evals.json") + workspace="/tmp/${skill}-eval-pr" + mkdir -p "${workspace}" + + echo "=== Running PR evals for ${skill} (${eval_count} cases) ===" + echo "Workspace: ${workspace}" + + claude -p "$(cat <&1 || { + echo "::warning::PR eval run failed for ${skill} (exit $?)" + continue + } + + echo "--- Workspace contents ---" + find "${workspace}" -type f 2>/dev/null | head -80 || true + done + + - name: Post eval results review + env: + SKILLS_CSV: ${{ needs.discover.outputs.skills }} + PR_NUMBER: ${{ needs.discover.outputs.pr_number }} + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const skills = process.env.SKILLS_CSV.split(',').filter(Boolean); + const prNumber = parseInt(process.env.PR_NUMBER); + + let body = ''; + for (const skill of skills) { + const summaryPath = `/tmp/${skill}-eval-pr/summary.md`; + if (fs.existsSync(summaryPath)) { + body += fs.readFileSync(summaryPath, 'utf8'); + } else { + body += `### ${skill}\n\n> No results produced. See workflow logs.\n\n`; + } + } + + const pluginJson = JSON.parse(fs.readFileSync('plugins/sdlc-workflow/.claude-plugin/plugin.json', 'utf8')); + const version = pluginJson.version; + + body += '---\n'; + body += `*Generated by [sdlc-workflow/run-evals](plugins/sdlc-workflow/skills/run-evals) v${version}*\n`; + + const { data: reviews } = await github.rest.pulls.listReviews({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber + }); + const marker = '## Eval Results'; + const existing = reviews.find(r => + r.user?.login === 'github-actions[bot]' && r.body?.startsWith(marker) + ); + + if (existing) { + await github.rest.pulls.updateReview({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber, + review_id: existing.id, + body + }); + } else { + await github.rest.pulls.createReview({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber, + event: 'COMMENT', + body + }); + } +``` + +- [ ] **Step 2: Validate YAML syntax** + +Run: `python3 -c "import yaml; yaml.safe_load(open('.github/workflows/eval-pr-run.yml')); print('Valid YAML')"` + +Expected: `Valid YAML` + +- [ ] **Step 3: Commit** + +```bash +git add .github/workflows/eval-pr-run.yml +git commit -m "feat(ci): add eval-pr-run.yml for fork PR eval dispatch + +Stage 2 of the workflow_run dispatch pattern. Downloads PR metadata +from eval-pr.yml, gates untrusted authors via eval-protected environment, +then runs evals with secret access and posts the PR review." +``` + +--- + +### Task 3: Update existing spec with cross-reference + +Add a reference from the original eval CI spec to the fork dispatch spec. + +**Files:** +- Modify: `docs/specs/2026-04-21-eval-skills-ci-workflow-design.md` + +- [ ] **Step 1: Add fork dispatch reference** + +Add the following line at the end of the "Security Considerations" section (after line 168, before the "Out of Scope" heading): + +```markdown +- Fork PR secret access is handled via `workflow_run` dispatch — see [Eval PR Fork Dispatch](2026-05-07-eval-pr-fork-dispatch-design.md). +``` + +- [ ] **Step 2: Commit** + +```bash +git add docs/specs/2026-04-21-eval-skills-ci-workflow-design.md +git commit -m "docs(specs): add fork dispatch cross-reference to eval CI spec" +``` + +--- + +### Post-Implementation: GitHub Environment Setup + +After merging, create the `eval-protected` environment in the repository: + +1. Go to Settings > Environments > New environment +2. Name: `eval-protected` +3. Add required reviewers (repository maintainers) +4. No environment-specific secrets needed — existing repository-level secrets are used From 0bd527bd5959161f69b4961d7d0fff274e29c733 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 10:11:41 +0200 Subject: [PATCH 03/18] refactor(ci): strip eval-pr.yml to discovery-only with artifact upload Stage 1 of the workflow_run dispatch pattern for fork PR secret access. Eval execution moves to eval-pr-run.yml (next commit). Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr.yml | 119 ++++++---------------------------- 1 file changed, 20 insertions(+), 99 deletions(-) diff --git a/.github/workflows/eval-pr.yml b/.github/workflows/eval-pr.yml index 989941a27..fda9d26b8 100644 --- a/.github/workflows/eval-pr.yml +++ b/.github/workflows/eval-pr.yml @@ -1,7 +1,7 @@ -# Runs skill evals on PRs that modify skill or eval files, compares results -# against the stored baseline, and posts a benchmark delta as a PR review. +# Discovers changed skills for PR evals and uploads metadata for the +# dispatch workflow (eval-pr-run.yml) to execute with secret access. # -# See docs/specs/2026-04-21-eval-skills-ci-workflow-design.md for full design. +# See docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md for full design. name: Eval PR @@ -14,26 +14,17 @@ on: - '.github/workflows/eval-pr.yml' jobs: - eval-pr: - name: Eval PR Changes + discover: + name: Discover Changed Skills runs-on: ubuntu-latest permissions: contents: read - pull-requests: write steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 0 - - name: Authenticate to Google Cloud - uses: google-github-actions/auth@v3 - with: - credentials_json: ${{ secrets.GCP_SA_KEY }} - - - name: Install Claude Code - run: curl -fsSL https://claude.ai/install.sh | bash - - name: Discover changed skills id: discover env: @@ -62,91 +53,21 @@ jobs: echo "skills=${skills}" >> "$GITHUB_OUTPUT" echo "Discovered changed skills with evals: ${skills:-none}" - - name: Run PR evals - if: steps.discover.outputs.skills != '' + - name: Write PR metadata env: - CLAUDE_CODE_USE_VERTEX: "1" - CLOUD_ML_REGION: ${{ secrets.GCP_CLOUD_ML_REGION }} - ANTHROPIC_DEFAULT_SONNET_MODEL: "claude-sonnet-4-6" - ANTHROPIC_DEFAULT_OPUS_MODEL: "claude-opus-4-6" - ANTHROPIC_MODEL: "claude-opus-4-6" + SKILLS: ${{ steps.discover.outputs.skills }} run: | - IFS=',' read -ra SKILLS <<< "${{ steps.discover.outputs.skills }}" - - for skill in "${SKILLS[@]}"; do - eval_count=$(jq '.evals | length' "evals/${skill}/evals.json") - workspace="/tmp/${skill}-eval-pr" - mkdir -p "${workspace}" - - echo "=== Running PR evals for ${skill} (${eval_count} cases) ===" - echo "Workspace: ${workspace}" - - claude -p "$(cat <&1 || { - echo "::warning::PR eval run failed for ${skill} (exit $?)" - continue - } - - echo "--- Workspace contents ---" - find "${workspace}" -type f 2>/dev/null | head -80 || true - done - - - name: Post eval results review - if: steps.discover.outputs.skills != '' - env: - SKILLS_CSV: ${{ steps.discover.outputs.skills }} - uses: actions/github-script@v7 + jq -n \ + --arg skills "$SKILLS" \ + --argjson pr_number ${{ github.event.pull_request.number }} \ + --arg head_sha "${{ github.event.pull_request.head.sha }}" \ + --arg author "${{ github.event.pull_request.user.login }}" \ + '{skills: $skills, pr_number: $pr_number, head_sha: $head_sha, author: $author}' \ + > eval-pr-metadata.json + cat eval-pr-metadata.json + + - name: Upload PR metadata + uses: actions/upload-artifact@v4 with: - script: | - const fs = require('fs'); - const skills = process.env.SKILLS_CSV.split(',').filter(Boolean); - - let body = ''; - for (const skill of skills) { - const summaryPath = `/tmp/${skill}-eval-pr/summary.md`; - if (fs.existsSync(summaryPath)) { - body += fs.readFileSync(summaryPath, 'utf8'); - } else { - body += `### ${skill}\n\n> No results produced. See workflow logs.\n\n`; - } - } - - const pluginJson = JSON.parse(fs.readFileSync('plugins/sdlc-workflow/.claude-plugin/plugin.json', 'utf8')); - const version = pluginJson.version; - - body += '---\n'; - body += `*Generated by [sdlc-workflow/run-evals](plugins/sdlc-workflow/skills/run-evals) v${version}*\n`; - - const { data: reviews } = await github.rest.pulls.listReviews({ - owner: context.repo.owner, - repo: context.repo.repo, - pull_number: context.issue.number - }); - const marker = '## Eval Results'; - const existing = reviews.find(r => - r.user?.login === 'github-actions[bot]' && r.body?.startsWith(marker) - ); - - if (existing) { - await github.rest.pulls.updateReview({ - owner: context.repo.owner, - repo: context.repo.repo, - pull_number: context.issue.number, - review_id: existing.id, - body - }); - } else { - await github.rest.pulls.createReview({ - owner: context.repo.owner, - repo: context.repo.repo, - pull_number: context.issue.number, - event: 'COMMENT', - body - }); - } + name: eval-pr-metadata + path: eval-pr-metadata.json From de4e3b07433e5ea3516c1f2e58d9338e1f17a856 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 10:16:30 +0200 Subject: [PATCH 04/18] feat(ci): add eval-pr-run.yml for fork PR eval dispatch Stage 2 of the workflow_run dispatch pattern. Downloads PR metadata from eval-pr.yml, gates untrusted authors via eval-protected environment, then runs evals with secret access and posts the PR review. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 189 ++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 .github/workflows/eval-pr-run.yml diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml new file mode 100644 index 000000000..8467f081a --- /dev/null +++ b/.github/workflows/eval-pr-run.yml @@ -0,0 +1,189 @@ +# Runs evals for PRs via workflow_run dispatch, enabling secret access +# for fork PRs. Pairs with eval-pr.yml (Stage 1: discovery). +# +# Trust model: +# - Collaborators with write/admin permission: evals run automatically +# - External contributors: require manual approval via eval-protected environment +# +# See docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md for full design. + +name: Eval PR Run + +on: + workflow_run: + workflows: ["Eval PR"] + types: [completed] + +permissions: + contents: read + pull-requests: write + actions: read + +jobs: + discover: + name: Discover PR Context + if: github.event.workflow_run.conclusion == 'success' + runs-on: ubuntu-latest + outputs: + skills: ${{ steps.metadata.outputs.skills }} + pr_number: ${{ steps.metadata.outputs.pr_number }} + head_sha: ${{ steps.metadata.outputs.head_sha }} + author: ${{ steps.metadata.outputs.author }} + trusted: ${{ steps.metadata.outputs.trusted }} + steps: + - name: Download PR metadata + uses: actions/download-artifact@v4 + with: + name: eval-pr-metadata + run-id: ${{ github.event.workflow_run.id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + + - name: Parse metadata and check trust + id: metadata + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const metadata = JSON.parse(fs.readFileSync('eval-pr-metadata.json', 'utf8')); + + core.setOutput('skills', metadata.skills); + core.setOutput('pr_number', metadata.pr_number.toString()); + core.setOutput('head_sha', metadata.head_sha); + core.setOutput('author', metadata.author); + + let trusted = false; + try { + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username: metadata.author + }); + trusted = ['admin', 'write'].includes(data.permission); + console.log(`Author ${metadata.author}: permission=${data.permission}, trusted=${trusted}`); + } catch (e) { + console.log(`Author ${metadata.author}: not a collaborator (${e.status}), trusted=false`); + } + + core.setOutput('trusted', trusted.toString()); + + gate: + name: Approval Gate + needs: discover + if: needs.discover.outputs.trusted != 'true' && needs.discover.outputs.skills != '' + runs-on: ubuntu-latest + environment: eval-protected + steps: + - run: echo "Approved by reviewer" + + run-evals: + name: Run PR Evals + needs: [discover, gate] + if: >- + !cancelled() && + needs.discover.result == 'success' && + needs.discover.outputs.skills != '' && + (needs.discover.outputs.trusted == 'true' || needs.gate.result == 'success') + runs-on: ubuntu-latest + steps: + - name: Checkout PR merge commit + uses: actions/checkout@v4 + with: + ref: refs/pull/${{ needs.discover.outputs.pr_number }}/merge + fetch-depth: 0 + + - name: Authenticate to Google Cloud + uses: google-github-actions/auth@v3 + with: + credentials_json: ${{ secrets.GCP_SA_KEY }} + + - name: Install Claude Code + run: curl -fsSL https://claude.ai/install.sh | bash + + - name: Run PR evals + env: + CLAUDE_CODE_USE_VERTEX: "1" + CLOUD_ML_REGION: ${{ secrets.GCP_CLOUD_ML_REGION }} + ANTHROPIC_DEFAULT_SONNET_MODEL: "claude-sonnet-4-6" + ANTHROPIC_DEFAULT_OPUS_MODEL: "claude-opus-4-6" + ANTHROPIC_MODEL: "claude-opus-4-6" + run: | + IFS=',' read -ra SKILLS <<< "${{ needs.discover.outputs.skills }}" + + for skill in "${SKILLS[@]}"; do + eval_count=$(jq '.evals | length' "evals/${skill}/evals.json") + workspace="/tmp/${skill}-eval-pr" + mkdir -p "${workspace}" + + echo "=== Running PR evals for ${skill} (${eval_count} cases) ===" + echo "Workspace: ${workspace}" + + claude -p "$(cat <&1 || { + echo "::warning::PR eval run failed for ${skill} (exit $?)" + continue + } + + echo "--- Workspace contents ---" + find "${workspace}" -type f 2>/dev/null | head -80 || true + done + + - name: Post eval results review + env: + SKILLS_CSV: ${{ needs.discover.outputs.skills }} + PR_NUMBER: ${{ needs.discover.outputs.pr_number }} + uses: actions/github-script@v7 + with: + script: | + const fs = require('fs'); + const skills = process.env.SKILLS_CSV.split(',').filter(Boolean); + const prNumber = parseInt(process.env.PR_NUMBER); + + let body = '## Eval Results\n\n'; + for (const skill of skills) { + const summaryPath = `/tmp/${skill}-eval-pr/summary.md`; + if (fs.existsSync(summaryPath)) { + body += fs.readFileSync(summaryPath, 'utf8'); + } else { + body += `### ${skill}\n\n> No results produced. See workflow logs.\n\n`; + } + } + + const pluginJson = JSON.parse(fs.readFileSync('plugins/sdlc-workflow/.claude-plugin/plugin.json', 'utf8')); + const version = pluginJson.version; + + body += '---\n'; + body += `*Generated by [sdlc-workflow/run-evals](plugins/sdlc-workflow/skills/run-evals) v${version}*\n`; + + const { data: reviews } = await github.rest.pulls.listReviews({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber + }); + const marker = '## Eval Results'; + const existing = reviews.find(r => + r.user?.login === 'github-actions[bot]' && r.body?.startsWith(marker) + ); + + if (existing) { + await github.rest.pulls.updateReview({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber, + review_id: existing.id, + body + }); + } else { + await github.rest.pulls.createReview({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber, + event: 'COMMENT', + body + }); + } From a712d4245c875858bd8b5a8ff9ade1d7cc27cdb2 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 10:20:51 +0200 Subject: [PATCH 05/18] docs(specs): add fork dispatch cross-reference to eval CI spec Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 5 ++++- docs/specs/2026-04-21-eval-skills-ci-workflow-design.md | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index 8467f081a..b2d502aee 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -69,7 +69,10 @@ jobs: gate: name: Approval Gate needs: discover - if: needs.discover.outputs.trusted != 'true' && needs.discover.outputs.skills != '' + if: >- + needs.discover.result == 'success' && + needs.discover.outputs.trusted != 'true' && + needs.discover.outputs.skills != '' runs-on: ubuntu-latest environment: eval-protected steps: diff --git a/docs/specs/2026-04-21-eval-skills-ci-workflow-design.md b/docs/specs/2026-04-21-eval-skills-ci-workflow-design.md index ff0771609..136b1a26f 100644 --- a/docs/specs/2026-04-21-eval-skills-ci-workflow-design.md +++ b/docs/specs/2026-04-21-eval-skills-ci-workflow-design.md @@ -166,6 +166,7 @@ The prompt describes the task in natural language (slash commands are interactiv - No secrets are exposed beyond `ANTHROPIC_API_KEY` - This is a feasibility proof, not a production gate - Future hardening: migrate to `--permission-mode dontAsk` with `--allowedTools` once tool requirements are known from initial runs. +- Fork PR secret access is handled via `workflow_run` dispatch — see [Eval PR Fork Dispatch](2026-05-07-eval-pr-fork-dispatch-design.md). ## Out of Scope From 3569a1e62c928d375c9d75f4dd3b413300c0fa1e Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 11:01:33 +0200 Subject: [PATCH 06/18] fix(ci): harden eval dispatch against artifact poisoning and injection Security fixes from review: - Derive PR number and author from GitHub API instead of artifact (prevents trust gate bypass via modified eval-pr.yml in fork PRs) - Validate skills format against [a-z0-9-] allowlist - Pass artifact-sourced values through env vars, not ${{ }} interpolation (prevents shell injection) - Remove --verbose from Claude invocation (prevents secret leakage in logs) - Add branches: [main] filter to workflow_run trigger Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 46 ++++++++++++++++++++++++------- .github/workflows/eval-pr.yml | 9 ++++-- 2 files changed, 42 insertions(+), 13 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index b2d502aee..1ebb68ec0 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -13,6 +13,7 @@ on: workflow_run: workflows: ["Eval PR"] types: [completed] + branches: [main] permissions: contents: read @@ -38,7 +39,7 @@ jobs: run-id: ${{ github.event.workflow_run.id }} github-token: ${{ secrets.GITHUB_TOKEN }} - - name: Parse metadata and check trust + - name: Resolve PR identity and check trust id: metadata uses: actions/github-script@v7 with: @@ -46,22 +47,46 @@ jobs: const fs = require('fs'); const metadata = JSON.parse(fs.readFileSync('eval-pr-metadata.json', 'utf8')); - core.setOutput('skills', metadata.skills); - core.setOutput('pr_number', metadata.pr_number.toString()); - core.setOutput('head_sha', metadata.head_sha); - core.setOutput('author', metadata.author); + // Validate skills format (artifact is untrusted — attacker can modify eval-pr.yml) + if (metadata.skills && !/^[a-z0-9-]+(,[a-z0-9-]+)*$/.test(metadata.skills)) { + core.setFailed(`Invalid skills format: ${metadata.skills}`); + return; + } + core.setOutput('skills', metadata.skills || ''); + + // Derive PR number and author from GitHub API to prevent spoofing. + // workflow_run.pull_requests is empty for fork PRs, so search by commit SHA. + const headSha = context.payload.workflow_run.head_sha; + const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ + owner: context.repo.owner, + repo: context.repo.repo, + commit_sha: headSha + }); + + if (prs.length === 0) { + core.setFailed(`No PR found for commit ${headSha}`); + return; + } + + const pr = prs[0]; + const prNumber = pr.number; + const author = pr.user.login; + + core.setOutput('pr_number', prNumber.toString()); + core.setOutput('head_sha', headSha); + core.setOutput('author', author); let trusted = false; try { const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ owner: context.repo.owner, repo: context.repo.repo, - username: metadata.author + username: author }); trusted = ['admin', 'write'].includes(data.permission); - console.log(`Author ${metadata.author}: permission=${data.permission}, trusted=${trusted}`); + console.log(`Author ${author}: permission=${data.permission}, trusted=${trusted}`); } catch (e) { - console.log(`Author ${metadata.author}: not a collaborator (${e.status}), trusted=false`); + console.log(`Author ${author}: not a collaborator (${e.status}), trusted=false`); } core.setOutput('trusted', trusted.toString()); @@ -104,13 +129,14 @@ jobs: - name: Run PR evals env: + SKILLS_CSV: ${{ needs.discover.outputs.skills }} CLAUDE_CODE_USE_VERTEX: "1" CLOUD_ML_REGION: ${{ secrets.GCP_CLOUD_ML_REGION }} ANTHROPIC_DEFAULT_SONNET_MODEL: "claude-sonnet-4-6" ANTHROPIC_DEFAULT_OPUS_MODEL: "claude-opus-4-6" ANTHROPIC_MODEL: "claude-opus-4-6" run: | - IFS=',' read -ra SKILLS <<< "${{ needs.discover.outputs.skills }}" + IFS=',' read -ra SKILLS <<< "$SKILLS_CSV" for skill in "${SKILLS[@]}"; do eval_count=$(jq '.evals | length' "evals/${skill}/evals.json") @@ -127,7 +153,7 @@ jobs: PROMPT )" --permission-mode dontAsk \ --allowedTools Read Write Bash Skill Agent Glob \ - --verbose 2>&1 || { + 2>&1 || { echo "::warning::PR eval run failed for ${skill} (exit $?)" continue } diff --git a/.github/workflows/eval-pr.yml b/.github/workflows/eval-pr.yml index fda9d26b8..0fc0ac398 100644 --- a/.github/workflows/eval-pr.yml +++ b/.github/workflows/eval-pr.yml @@ -56,12 +56,15 @@ jobs: - name: Write PR metadata env: SKILLS: ${{ steps.discover.outputs.skills }} + PR_NUMBER: ${{ github.event.pull_request.number }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + AUTHOR: ${{ github.event.pull_request.user.login }} run: | jq -n \ --arg skills "$SKILLS" \ - --argjson pr_number ${{ github.event.pull_request.number }} \ - --arg head_sha "${{ github.event.pull_request.head.sha }}" \ - --arg author "${{ github.event.pull_request.user.login }}" \ + --argjson pr_number "$PR_NUMBER" \ + --arg head_sha "$HEAD_SHA" \ + --arg author "$AUTHOR" \ '{skills: $skills, pr_number: $pr_number, head_sha: $head_sha, author: $author}' \ > eval-pr-metadata.json cat eval-pr-metadata.json From b219c160f76ae98399d181dbe0470d67371aba6e Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 12:58:53 +0200 Subject: [PATCH 07/18] fix(ci): cross-validate PR number and remove unused head_sha output Use artifact's pr_number to disambiguate when multiple PRs share a commit. Remove head_sha from job outputs (never consumed downstream). Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index 1ebb68ec0..5f51e65c0 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -28,7 +28,6 @@ jobs: outputs: skills: ${{ steps.metadata.outputs.skills }} pr_number: ${{ steps.metadata.outputs.pr_number }} - head_sha: ${{ steps.metadata.outputs.head_sha }} author: ${{ steps.metadata.outputs.author }} trusted: ${{ steps.metadata.outputs.trusted }} steps: @@ -68,12 +67,11 @@ jobs: return; } - const pr = prs[0]; + const pr = prs.find(p => p.number === metadata.pr_number) || prs[0]; const prNumber = pr.number; const author = pr.user.login; core.setOutput('pr_number', prNumber.toString()); - core.setOutput('head_sha', headSha); core.setOutput('author', author); let trusted = false; From d3cc5f9017e79d28ca0dfe5692053f26ad8815b4 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 13:13:11 +0200 Subject: [PATCH 08/18] fix(ci): remove branches filter and drop author from artifact MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit branches: [main] on workflow_run filters on the PR's head branch, not the base branch — confirmed broken by PR #123 (Stage 2 never triggered). Remove it; the triggering workflow's own filter already restricts to PRs targeting main. Also remove author from artifact metadata since it's derived from the GitHub API in Stage 2 for security. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 2 +- .github/workflows/eval-pr.yml | 4 +--- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index 5f51e65c0..b6fbd1937 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -13,7 +13,7 @@ on: workflow_run: workflows: ["Eval PR"] types: [completed] - branches: [main] + permissions: contents: read diff --git a/.github/workflows/eval-pr.yml b/.github/workflows/eval-pr.yml index 0fc0ac398..72fb9f5ba 100644 --- a/.github/workflows/eval-pr.yml +++ b/.github/workflows/eval-pr.yml @@ -58,14 +58,12 @@ jobs: SKILLS: ${{ steps.discover.outputs.skills }} PR_NUMBER: ${{ github.event.pull_request.number }} HEAD_SHA: ${{ github.event.pull_request.head.sha }} - AUTHOR: ${{ github.event.pull_request.user.login }} run: | jq -n \ --arg skills "$SKILLS" \ --argjson pr_number "$PR_NUMBER" \ --arg head_sha "$HEAD_SHA" \ - --arg author "$AUTHOR" \ - '{skills: $skills, pr_number: $pr_number, head_sha: $head_sha, author: $author}' \ + '{skills: $skills, pr_number: $pr_number, head_sha: $head_sha}' \ > eval-pr-metadata.json cat eval-pr-metadata.json From 206a78d3e5a10b7f0e8de5aa7473b3998e12664c Mon Sep 17 00:00:00 2001 From: mrizzi Date: Thu, 7 May 2026 13:18:41 +0200 Subject: [PATCH 09/18] fix(ci): remove artifact influence on PR selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Select PR solely from GitHub API (open PRs targeting main for the commit SHA). Artifact now carries only the skills CSV — no identity fields that could influence routing. Addresses Sourcery review feedback on PR #123. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 7 ++++--- .github/workflows/eval-pr.yml | 6 +----- 2 files changed, 5 insertions(+), 8 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index b6fbd1937..e6de81814 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -62,12 +62,13 @@ jobs: commit_sha: headSha }); - if (prs.length === 0) { - core.setFailed(`No PR found for commit ${headSha}`); + const openToMain = prs.filter(p => p.state === 'open' && p.base.ref === 'main'); + if (openToMain.length === 0) { + core.setFailed(`No open PR targeting main found for commit ${headSha}`); return; } - const pr = prs.find(p => p.number === metadata.pr_number) || prs[0]; + const pr = openToMain[0]; const prNumber = pr.number; const author = pr.user.login; diff --git a/.github/workflows/eval-pr.yml b/.github/workflows/eval-pr.yml index 72fb9f5ba..f718eaf10 100644 --- a/.github/workflows/eval-pr.yml +++ b/.github/workflows/eval-pr.yml @@ -56,14 +56,10 @@ jobs: - name: Write PR metadata env: SKILLS: ${{ steps.discover.outputs.skills }} - PR_NUMBER: ${{ github.event.pull_request.number }} - HEAD_SHA: ${{ github.event.pull_request.head.sha }} run: | jq -n \ --arg skills "$SKILLS" \ - --argjson pr_number "$PR_NUMBER" \ - --arg head_sha "$HEAD_SHA" \ - '{skills: $skills, pr_number: $pr_number, head_sha: $head_sha}' \ + '{skills: $skills}' \ > eval-pr-metadata.json cat eval-pr-metadata.json From f6f627bb8c4a990e10274508cf9b101f74b62bd4 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 13:05:45 +0200 Subject: [PATCH 10/18] docs(specs): sync spec with implementation Update spec to reflect security hardening changes: - Artifact carries only skills (not pr_number, head_sha, author) - PR identity derived from GitHub API via listPullRequestsAssociatedWithCommit - Skills validated against allowlist regex - Gate and run-evals conditions include needs.discover.result check Co-Authored-By: Claude Opus 4.6 (1M context) --- ...2026-05-07-eval-pr-fork-dispatch-design.md | 55 ++++++++++--------- 1 file changed, 29 insertions(+), 26 deletions(-) diff --git a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md index ceab41c94..57d57815e 100644 --- a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md +++ b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md @@ -43,8 +43,7 @@ Fork PR opened │ eval-pr.yml (pull_request) │ No secrets needed │ 1. Checkout │ │ 2. Discover changed skills │ -│ 3. Upload artifact │ -│ (skills, PR#, SHA, author)│ +│ 3. Upload artifact (skills) │ └──────────────┬───────────────┘ │ workflow_run (completed) ▼ @@ -52,9 +51,10 @@ Fork PR opened │ eval-pr-run.yml (workflow_run) │ Base repo context — secrets available │ │ │ Job 1: discover │ -│ - Download artifact │ +│ - Download artifact (skills only) │ +│ - Resolve PR identity via GitHub API│ │ - Check collaborator permission │ -│ - Output: trusted, skills, PR#, SHA │ +│ - Output: trusted, skills, PR# │ │ │ │ Job 2: gate │ │ - Runs ONLY if trusted != true │ @@ -102,17 +102,16 @@ Reduced from `contents: read` + `pull-requests: write` — this workflow no long 1. **Checkout** — `actions/checkout@v4` with `fetch-depth: 0` 2. **Discover changed skills** — existing logic, unchanged (git diff against PR base, map to eval suites) -3. **Write metadata file** — write `eval-pr-metadata.json` with PR context: +3. **Write metadata file** — write `eval-pr-metadata.json` with discovered skills: ```json { - "skills": "skill1,skill2", - "pr_number": 90, - "head_sha": "abc123def", - "author": "username" + "skills": "skill1,skill2" } ``` +The artifact carries only the discovery result. PR identity (number, author) is derived from the GitHub API in Stage 2 to prevent spoofing via artifact poisoning. + 4. **Upload artifact** — `actions/upload-artifact@v4` with name `eval-pr-metadata`, containing the `eval-pr-metadata.json` file. The artifact is uploaded even when `skills` is empty so Stage 2 can detect the no-op cleanly. @@ -159,33 +158,34 @@ Downloads the artifact and determines trust level. Skips early if the triggering github-token: ${{ secrets.GITHUB_TOKEN }} ``` -3. **Parse metadata and check trust** via `actions/github-script@v7`: +3. **Validate skills and resolve PR identity** via `actions/github-script@v7`: + - Validate `skills` format against `^[a-z0-9-]+(,[a-z0-9-]+)*$` (artifact is untrusted) + - Resolve PR number and author from GitHub API via `listPullRequestsAssociatedWithCommit` using `workflow_run.head_sha`, filtered to open PRs targeting `main` + - Check collaborator permission level for the API-derived author ```javascript -const metadata = JSON.parse(fs.readFileSync('eval-pr-metadata.json', 'utf8')); - -let trusted = false; -try { - const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ - owner: context.repo.owner, - repo: context.repo.repo, - username: metadata.author - }); - trusted = ['admin', 'write'].includes(data.permission); -} catch (e) { - // Non-collaborators may 404 — treat as untrusted - trusted = false; -} +const headSha = context.payload.workflow_run.head_sha; +const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ + owner: context.repo.owner, repo: context.repo.repo, commit_sha: headSha +}); +const pr = prs.filter(p => p.state === 'open' && p.base.ref === 'main')[0]; +const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, repo: context.repo.repo, username: pr.user.login +}); +const trusted = ['admin', 'write'].includes(data.permission); ``` -4. **Set outputs**: `skills`, `pr_number`, `head_sha`, `author`, `trusted` +4. **Set outputs**: `skills`, `pr_number`, `author`, `trusted` ### Job 2: gate ```yaml gate: needs: discover - if: needs.discover.outputs.trusted != 'true' && needs.discover.outputs.skills != '' + if: >- + needs.discover.result == 'success' && + needs.discover.outputs.trusted != 'true' && + needs.discover.outputs.skills != '' runs-on: ubuntu-latest environment: eval-protected steps: @@ -201,6 +201,7 @@ run-evals: needs: [discover, gate] if: >- !cancelled() && + needs.discover.result == 'success' && needs.discover.outputs.skills != '' && (needs.discover.outputs.trusted == 'true' || needs.gate.result == 'success') runs-on: ubuntu-latest @@ -250,6 +251,8 @@ All other PR authors. Their eval runs require a reviewer to click "Approve" in t ### Security Properties - The `workflow_run` workflow executes on the base repository's default branch — a fork PR cannot modify the workflow definition or trust logic +- PR identity (number, author) is derived from the GitHub API via `listPullRequestsAssociatedWithCommit`, not from the artifact — prevents spoofing via artifact poisoning +- The artifact carries only the `skills` CSV, validated against an allowlist regex — it has no influence on PR routing or trust decisions - The collaborator permission check uses GitHub's API, not a file in the repo — it cannot be manipulated by a PR - Once approved, the workflow does execute skill files from the PR with access to secrets via environment variables — this is inherent to the use case (evaluating skills requires running them) - The `dontAsk` permission mode with explicit `--allowedTools` limits Claude Code's capabilities, but a malicious skill could still read environment variables via the Bash tool From fb1ce73272bdc6693c169aad2a76e670a53a2104 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 13:13:32 +0200 Subject: [PATCH 11/18] =?UTF-8?q?refactor(ci):=20eliminate=20artifact=20?= =?UTF-8?q?=E2=80=94=20discover=20skills=20from=20checkout?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 1 (eval-pr.yml) is now a minimal path-filtered trigger with no artifact. Stage 2 (eval-pr-run.yml) derives everything from the GitHub API and the checked-out PR merge commit: PR identity from the API, skills from git diff on the checkout. No data from Stage 1 is trusted. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 84 ++++++++++++++++++------------- .github/workflows/eval-pr.yml | 58 +++------------------ 2 files changed, 56 insertions(+), 86 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index e6de81814..da42d71ce 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -1,10 +1,14 @@ # Runs evals for PRs via workflow_run dispatch, enabling secret access -# for fork PRs. Pairs with eval-pr.yml (Stage 1: discovery). +# for fork PRs. Triggered by eval-pr.yml completion (path-filtered gate). # # Trust model: # - Collaborators with write/admin permission: evals run automatically # - External contributors: require manual approval via eval-protected environment # +# All PR identity and skill discovery is derived from the GitHub API and +# the checked-out PR merge commit — no data from the triggering workflow +# is trusted. +# # See docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md for full design. name: Eval PR Run @@ -14,7 +18,6 @@ on: workflows: ["Eval PR"] types: [completed] - permissions: contents: read pull-requests: write @@ -26,35 +29,16 @@ jobs: if: github.event.workflow_run.conclusion == 'success' runs-on: ubuntu-latest outputs: - skills: ${{ steps.metadata.outputs.skills }} - pr_number: ${{ steps.metadata.outputs.pr_number }} - author: ${{ steps.metadata.outputs.author }} - trusted: ${{ steps.metadata.outputs.trusted }} + skills: ${{ steps.discover.outputs.skills }} + pr_number: ${{ steps.pr.outputs.pr_number }} + author: ${{ steps.pr.outputs.author }} + trusted: ${{ steps.pr.outputs.trusted }} steps: - - name: Download PR metadata - uses: actions/download-artifact@v4 - with: - name: eval-pr-metadata - run-id: ${{ github.event.workflow_run.id }} - github-token: ${{ secrets.GITHUB_TOKEN }} - - name: Resolve PR identity and check trust - id: metadata + id: pr uses: actions/github-script@v7 with: script: | - const fs = require('fs'); - const metadata = JSON.parse(fs.readFileSync('eval-pr-metadata.json', 'utf8')); - - // Validate skills format (artifact is untrusted — attacker can modify eval-pr.yml) - if (metadata.skills && !/^[a-z0-9-]+(,[a-z0-9-]+)*$/.test(metadata.skills)) { - core.setFailed(`Invalid skills format: ${metadata.skills}`); - return; - } - core.setOutput('skills', metadata.skills || ''); - - // Derive PR number and author from GitHub API to prevent spoofing. - // workflow_run.pull_requests is empty for fork PRs, so search by commit SHA. const headSha = context.payload.workflow_run.head_sha; const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ owner: context.repo.owner, @@ -69,27 +53,59 @@ jobs: } const pr = openToMain[0]; - const prNumber = pr.number; - const author = pr.user.login; - - core.setOutput('pr_number', prNumber.toString()); - core.setOutput('author', author); + core.setOutput('pr_number', pr.number.toString()); + core.setOutput('base_sha', pr.base.sha); + core.setOutput('author', pr.user.login); let trusted = false; try { const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ owner: context.repo.owner, repo: context.repo.repo, - username: author + username: pr.user.login }); trusted = ['admin', 'write'].includes(data.permission); - console.log(`Author ${author}: permission=${data.permission}, trusted=${trusted}`); + console.log(`Author ${pr.user.login}: permission=${data.permission}, trusted=${trusted}`); } catch (e) { - console.log(`Author ${author}: not a collaborator (${e.status}), trusted=false`); + console.log(`Author ${pr.user.login}: not a collaborator (${e.status}), trusted=false`); } core.setOutput('trusted', trusted.toString()); + - name: Checkout PR merge commit + uses: actions/checkout@v4 + with: + ref: refs/pull/${{ steps.pr.outputs.pr_number }}/merge + fetch-depth: 0 + + - name: Discover changed skills + id: discover + env: + BASE_SHA: ${{ steps.pr.outputs.base_sha }} + run: | + changed_files=$(git diff --name-only "$BASE_SHA"...HEAD) + echo "Changed files:" + echo "$changed_files" + + skills="" + for file in $changed_files; do + skill="" + if [[ "$file" =~ ^plugins/sdlc-workflow/skills/([^/]+)/ ]]; then + skill="${BASH_REMATCH[1]}" + elif [[ "$file" =~ ^evals/([^/]+)/ ]]; then + skill="${BASH_REMATCH[1]}" + fi + + if [ -n "$skill" ] && [ -f "evals/${skill}/evals.json" ]; then + if [[ ! ",$skills," == *",$skill,"* ]]; then + skills="${skills:+${skills},}${skill}" + fi + fi + done + + echo "skills=${skills}" >> "$GITHUB_OUTPUT" + echo "Discovered changed skills with evals: ${skills:-none}" + gate: name: Approval Gate needs: discover diff --git a/.github/workflows/eval-pr.yml b/.github/workflows/eval-pr.yml index f718eaf10..cc676c384 100644 --- a/.github/workflows/eval-pr.yml +++ b/.github/workflows/eval-pr.yml @@ -1,5 +1,6 @@ -# Discovers changed skills for PR evals and uploads metadata for the -# dispatch workflow (eval-pr-run.yml) to execute with secret access. +# Path-filtered trigger for the eval dispatch workflow (eval-pr-run.yml). +# This workflow does no work itself — its completion signals eval-pr-run.yml +# to run via workflow_run. # # See docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md for full design. @@ -14,57 +15,10 @@ on: - '.github/workflows/eval-pr.yml' jobs: - discover: - name: Discover Changed Skills + trigger: + name: Trigger Eval Dispatch runs-on: ubuntu-latest permissions: contents: read steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Discover changed skills - id: discover - env: - BASE_SHA: ${{ github.event.pull_request.base.sha }} - run: | - changed_files=$(git diff --name-only "$BASE_SHA"...HEAD) - echo "Changed files:" - echo "$changed_files" - - skills="" - for file in $changed_files; do - skill="" - if [[ "$file" =~ ^plugins/sdlc-workflow/skills/([^/]+)/ ]]; then - skill="${BASH_REMATCH[1]}" - elif [[ "$file" =~ ^evals/([^/]+)/ ]]; then - skill="${BASH_REMATCH[1]}" - fi - - if [ -n "$skill" ] && [ -f "evals/${skill}/evals.json" ]; then - if [[ ! ",$skills," == *",$skill,"* ]]; then - skills="${skills:+${skills},}${skill}" - fi - fi - done - - echo "skills=${skills}" >> "$GITHUB_OUTPUT" - echo "Discovered changed skills with evals: ${skills:-none}" - - - name: Write PR metadata - env: - SKILLS: ${{ steps.discover.outputs.skills }} - run: | - jq -n \ - --arg skills "$SKILLS" \ - '{skills: $skills}' \ - > eval-pr-metadata.json - cat eval-pr-metadata.json - - - name: Upload PR metadata - uses: actions/upload-artifact@v4 - with: - name: eval-pr-metadata - path: eval-pr-metadata.json + - run: echo "PR touches eval paths — dispatching eval run" From 7f0efc0ff73f3b51962749c97005e98de2120d9a Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 13:15:27 +0200 Subject: [PATCH 12/18] docs(specs): sync spec with artifact-free implementation Remove all artifact references. Stage 1 is now a minimal trigger, Stage 2 derives everything from the GitHub API and git diff. Co-Authored-By: Claude Opus 4.6 (1M context) --- ...2026-05-07-eval-pr-fork-dispatch-design.md | 75 +++++-------------- 1 file changed, 20 insertions(+), 55 deletions(-) diff --git a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md index 57d57815e..3be900dec 100644 --- a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md +++ b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md @@ -31,7 +31,7 @@ A trust check auto-approves eval runs for repository collaborators with write ac | Trust threshold | `admin` or `write` | Collaborators with write access can already push to the repo directly — granting them secret access in CI adds no new attack surface | | Untrusted gate | Protected environment | GitHub's built-in approval mechanism — reviewers click "Approve" in the Actions UI | | PR coverage | All PRs (fork and non-fork) | Single code path avoids duplicate logic and ensures consistent behavior | -| Artifact naming | `eval-pr-metadata` (fixed name) | Artifacts are scoped to workflow runs — concurrent PRs each get their own run ID, no collision | +| Data flow | No artifact — Stage 2 derives everything | Artifact is attacker-controlled (fork PR can modify Stage 1 workflow). Stage 2 derives PR identity from GitHub API and skills from git diff on the checked-out merge commit | ## Architecture @@ -40,10 +40,9 @@ Fork PR opened │ ▼ ┌──────────────────────────────┐ -│ eval-pr.yml (pull_request) │ No secrets needed -│ 1. Checkout │ -│ 2. Discover changed skills │ -│ 3. Upload artifact (skills) │ +│ eval-pr.yml (pull_request) │ Path-filtered gate only +│ No work — just triggers │ +│ workflow_run on completion │ └──────────────┬───────────────┘ │ workflow_run (completed) ▼ @@ -51,8 +50,9 @@ Fork PR opened │ eval-pr-run.yml (workflow_run) │ Base repo context — secrets available │ │ │ Job 1: discover │ -│ - Download artifact (skills only) │ │ - Resolve PR identity via GitHub API│ +│ - Checkout PR merge commit │ +│ - Discover skills via git diff │ │ - Check collaborator permission │ │ - Output: trusted, skills, PR# │ │ │ @@ -73,7 +73,7 @@ Fork PR opened ### Changes from Current -The workflow is stripped to discovery and artifact upload only. All secret-dependent steps are removed: GCP authentication, Claude Code installation, eval execution, and review posting. +The workflow is reduced to a minimal path-filtered trigger. All logic is removed — no checkout, no discovery, no artifact. Its sole purpose is to act as a gate: GitHub's path filter ensures it only runs when skill or eval files change, and its completion triggers Stage 2 via `workflow_run`. ### Trigger @@ -89,32 +89,9 @@ on: - '.github/workflows/eval-pr.yml' ``` -### Permissions - -```yaml -permissions: - contents: read -``` - -Reduced from `contents: read` + `pull-requests: write` — this workflow no longer posts reviews. - ### Steps -1. **Checkout** — `actions/checkout@v4` with `fetch-depth: 0` -2. **Discover changed skills** — existing logic, unchanged (git diff against PR base, map to eval suites) -3. **Write metadata file** — write `eval-pr-metadata.json` with discovered skills: - -```json -{ - "skills": "skill1,skill2" -} -``` - -The artifact carries only the discovery result. PR identity (number, author) is derived from the GitHub API in Stage 2 to prevent spoofing via artifact poisoning. - -4. **Upload artifact** — `actions/upload-artifact@v4` with name `eval-pr-metadata`, containing the `eval-pr-metadata.json` file. - -The artifact is uploaded even when `skills` is empty so Stage 2 can detect the no-op cleanly. +Single no-op step. The workflow just needs to complete successfully to trigger Stage 2. ## Workflow 2: eval-pr-run.yml (new) @@ -127,7 +104,7 @@ on: types: [completed] ``` -No path filtering — `workflow_run` does not support it. The `completed` type fires on both success and failure, so the `discover` job checks `github.event.workflow_run.conclusion == 'success'` before proceeding. If the triggering workflow found no changed skills, the `run-evals` job skips via its `if` condition. +No path filtering — `workflow_run` does not support it. The `completed` type fires on both success and failure, so the `discover` job checks `github.event.workflow_run.conclusion == 'success'` before proceeding. ### Permissions @@ -135,33 +112,18 @@ No path filtering — `workflow_run` does not support it. The `completed` type f permissions: contents: read pull-requests: write - actions: read ``` -`actions: read` is needed to download artifacts from the triggering run. - ### Job 1: discover -Downloads the artifact and determines trust level. Skips early if the triggering workflow failed. +Resolves PR identity from the GitHub API, checks out the PR merge commit, discovers changed skills via git diff, and determines trust level. Skips early if the triggering workflow failed. No data from the triggering workflow is used — everything is derived independently. **Steps:** -1. **Check triggering workflow conclusion** — exit early if `github.event.workflow_run.conclusion != 'success'` - -2. **Download artifact** from triggering run: - -```yaml -- uses: actions/download-artifact@v4 - with: - name: eval-pr-metadata - run-id: ${{ github.event.workflow_run.id }} - github-token: ${{ secrets.GITHUB_TOKEN }} -``` - -3. **Validate skills and resolve PR identity** via `actions/github-script@v7`: - - Validate `skills` format against `^[a-z0-9-]+(,[a-z0-9-]+)*$` (artifact is untrusted) - - Resolve PR number and author from GitHub API via `listPullRequestsAssociatedWithCommit` using `workflow_run.head_sha`, filtered to open PRs targeting `main` - - Check collaborator permission level for the API-derived author +1. **Resolve PR identity and check trust** via `actions/github-script@v7`: + - Find PR via `listPullRequestsAssociatedWithCommit` using `workflow_run.head_sha`, filtered to open PRs targeting `main` + - Check collaborator permission level for the PR author + - Output `pr_number`, `base_sha`, `author`, `trusted` ```javascript const headSha = context.payload.workflow_run.head_sha; @@ -175,6 +137,10 @@ const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ const trusted = ['admin', 'write'].includes(data.permission); ``` +2. **Checkout PR merge commit** — `actions/checkout@v4` with `ref: refs/pull//merge` and `fetch-depth: 0` + +3. **Discover changed skills** — git diff against `base_sha`, same logic as the original `eval-pr.yml` discovery step (map changed files to eval suites) + 4. **Set outputs**: `skills`, `pr_number`, `author`, `trusted` ### Job 2: gate @@ -251,8 +217,7 @@ All other PR authors. Their eval runs require a reviewer to click "Approve" in t ### Security Properties - The `workflow_run` workflow executes on the base repository's default branch — a fork PR cannot modify the workflow definition or trust logic -- PR identity (number, author) is derived from the GitHub API via `listPullRequestsAssociatedWithCommit`, not from the artifact — prevents spoofing via artifact poisoning -- The artifact carries only the `skills` CSV, validated against an allowlist regex — it has no influence on PR routing or trust decisions +- No data passes from Stage 1 to Stage 2 — there is no artifact. Stage 2 derives everything independently: PR identity from the GitHub API, skills from git diff on the checked-out merge commit. This eliminates artifact poisoning as an attack vector entirely. - The collaborator permission check uses GitHub's API, not a file in the repo — it cannot be manipulated by a PR - Once approved, the workflow does execute skill files from the PR with access to secrets via environment variables — this is inherent to the use case (evaluating skills requires running them) - The `dontAsk` permission mode with explicit `--allowedTools` limits Claude Code's capabilities, but a malicious skill could still read environment variables via the Bash tool @@ -286,7 +251,7 @@ Create a protected environment in the repository: | File | Change | |------|--------| -| `.github/workflows/eval-pr.yml` | Strip to discovery + artifact upload only | +| `.github/workflows/eval-pr.yml` | Reduce to minimal path-filtered trigger (no discovery, no artifact) | | `.github/workflows/eval-pr-run.yml` | New workflow: trust check, gate, eval execution, review posting | | `docs/specs/2026-04-21-eval-skills-ci-workflow-design.md` | Add reference to this spec | From 2d1fb5286bb2977f46704f52b50ba328f27685b4 Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 16:47:50 +0200 Subject: [PATCH 13/18] fix(ci): use pulls.list for fork PR resolution listPullRequestsAssociatedWithCommit returns empty for fork PRs because the commit lives in the fork's repo, not the base repo's commit graph. Use pulls.list filtered by base branch and match by head.sha instead. Validated in https://github.com/mrizzi/test-workflow-dispact/actions/runs/25561566262 Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 15 +++++++++------ .../2026-05-07-eval-pr-fork-dispatch-design.md | 11 +++++++---- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index da42d71ce..9d4fa7c12 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -39,20 +39,23 @@ jobs: uses: actions/github-script@v7 with: script: | + // listPullRequestsAssociatedWithCommit returns empty for fork PRs + // because the commit lives in the fork's repo, not the base repo's + // commit graph. Use pulls.list filtered by base branch instead. const headSha = context.payload.workflow_run.head_sha; - const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ + const { data: prs } = await github.rest.pulls.list({ owner: context.repo.owner, repo: context.repo.repo, - commit_sha: headSha + state: 'open', + base: 'main', + per_page: 100 }); - const openToMain = prs.filter(p => p.state === 'open' && p.base.ref === 'main'); - if (openToMain.length === 0) { + const pr = prs.find(p => p.head.sha === headSha); + if (!pr) { core.setFailed(`No open PR targeting main found for commit ${headSha}`); return; } - - const pr = openToMain[0]; core.setOutput('pr_number', pr.number.toString()); core.setOutput('base_sha', pr.base.sha); core.setOutput('author', pr.user.login); diff --git a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md index 3be900dec..1adf90bcd 100644 --- a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md +++ b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md @@ -121,16 +121,19 @@ Resolves PR identity from the GitHub API, checks out the PR merge commit, discov **Steps:** 1. **Resolve PR identity and check trust** via `actions/github-script@v7`: - - Find PR via `listPullRequestsAssociatedWithCommit` using `workflow_run.head_sha`, filtered to open PRs targeting `main` + - Find PR via `pulls.list` filtered by `base: 'main'`, matched by `head.sha == workflow_run.head_sha` - Check collaborator permission level for the PR author - Output `pr_number`, `base_sha`, `author`, `trusted` +Note: `listPullRequestsAssociatedWithCommit` does not work for fork PRs because the commit lives in the fork's repository, not the base repo's commit graph. `pulls.list` with SHA matching is the reliable alternative. + ```javascript const headSha = context.payload.workflow_run.head_sha; -const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ - owner: context.repo.owner, repo: context.repo.repo, commit_sha: headSha +const { data: prs } = await github.rest.pulls.list({ + owner: context.repo.owner, repo: context.repo.repo, + state: 'open', base: 'main', per_page: 100 }); -const pr = prs.filter(p => p.state === 'open' && p.base.ref === 'main')[0]; +const pr = prs.find(p => p.head.sha === headSha); const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ owner: context.repo.owner, repo: context.repo.repo, username: pr.user.login }); From cd98189b006e0ac059ff274d95a006862a311eba Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 16:51:23 +0200 Subject: [PATCH 14/18] fix(ci): add pagination to pulls.list for PR resolution Use github.paginate to handle repos with many open PRs. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index 9d4fa7c12..a3b15f6f0 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -43,7 +43,7 @@ jobs: // because the commit lives in the fork's repo, not the base repo's // commit graph. Use pulls.list filtered by base branch instead. const headSha = context.payload.workflow_run.head_sha; - const { data: prs } = await github.rest.pulls.list({ + const prs = await github.paginate(github.rest.pulls.list, { owner: context.repo.owner, repo: context.repo.repo, state: 'open', From e5dbf594fe4b05f54beda9666a82ec7c463e337b Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 17:11:41 +0200 Subject: [PATCH 15/18] feat(ci): add commit status reporting for PR visibility workflow_run workflows don't appear as status checks on the PR page. Use the Commit Status API to post pending/success/failure statuses on the PR's head SHA so authors and reviewers have visibility without checking the Actions tab. Validated in https://github.com/mrizzi/test-workflow-dispact/pull/2 Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 70 +++++++++++++++++++ ...2026-05-07-eval-pr-fork-dispatch-design.md | 19 +++++ 2 files changed, 89 insertions(+) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index a3b15f6f0..4bae79b4b 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -22,6 +22,7 @@ permissions: contents: read pull-requests: write actions: read + statuses: write jobs: discover: @@ -34,6 +35,20 @@ jobs: author: ${{ steps.pr.outputs.author }} trusted: ${{ steps.pr.outputs.trusted }} steps: + - name: Set pending commit status + uses: actions/github-script@v7 + with: + script: | + await github.rest.repos.createCommitStatus({ + owner: context.repo.owner, + repo: context.repo.repo, + sha: context.payload.workflow_run.head_sha, + state: 'pending', + description: 'Eval run in progress...', + context: 'Eval PR Run', + target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}` + }); + - name: Resolve PR identity and check trust id: pr uses: actions/github-script@v7 @@ -109,6 +124,21 @@ jobs: echo "skills=${skills}" >> "$GITHUB_OUTPUT" echo "Discovered changed skills with evals: ${skills:-none}" + - name: Update status for approval gate + if: steps.pr.outputs.trusted != 'true' && steps.pr.outputs.pr_number != '' + uses: actions/github-script@v7 + with: + script: | + await github.rest.repos.createCommitStatus({ + owner: context.repo.owner, + repo: context.repo.repo, + sha: context.payload.workflow_run.head_sha, + state: 'pending', + description: 'Waiting for approval in eval-protected environment', + context: 'Eval PR Run', + target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}` + }); + gate: name: Approval Gate needs: discover @@ -234,3 +264,43 @@ jobs: body }); } + + report-status: + name: Report Status + needs: [discover, gate, run-evals] + if: >- + always() && + needs.discover.result == 'success' + runs-on: ubuntu-latest + steps: + - name: Set final commit status + env: + EVALS_RESULT: ${{ needs.run-evals.result }} + GATE_RESULT: ${{ needs.gate.result }} + uses: actions/github-script@v7 + with: + script: | + const evalsResult = process.env.EVALS_RESULT; + const gateResult = process.env.GATE_RESULT; + + let state, description; + if (evalsResult === 'success') { + state = 'success'; + description = 'Eval run completed — results posted as PR review'; + } else if (gateResult === 'failure') { + state = 'failure'; + description = 'Approval was rejected in eval-protected environment'; + } else { + state = 'failure'; + description = 'Eval run failed — check workflow logs'; + } + + await github.rest.repos.createCommitStatus({ + owner: context.repo.owner, + repo: context.repo.repo, + sha: context.payload.workflow_run.head_sha, + state, + description, + context: 'Eval PR Run', + target_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}` + }); diff --git a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md index 1adf90bcd..3768aef7a 100644 --- a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md +++ b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md @@ -50,10 +50,12 @@ Fork PR opened │ eval-pr-run.yml (workflow_run) │ Base repo context — secrets available │ │ │ Job 1: discover │ +│ - Set pending commit status on PR │ │ - Resolve PR identity via GitHub API│ │ - Checkout PR merge commit │ │ - Discover skills via git diff │ │ - Check collaborator permission │ +│ - Update status if awaiting approval│ │ - Output: trusted, skills, PR# │ │ │ │ Job 2: gate │ @@ -66,6 +68,9 @@ Fork PR opened │ - Authenticate GCP │ │ - Run evals │ │ - Post PR review │ +│ │ +│ Job 4: report-status │ +│ - Set final commit status (pass/fail)│ └──────────────────────────────────────┘ ``` @@ -201,6 +206,20 @@ run-evals: 5. **Post eval results review** — existing step, moved from `eval-pr.yml`. Uses `needs.discover.outputs.pr_number` instead of `context.issue.number` (which is unavailable in `workflow_run` context). +### Job 4: report-status + +Sets a final commit status on the PR's head SHA via the Commit Status API. Runs after all other jobs complete (`always()` + `needs.discover.result == 'success'`). + +This is necessary because `workflow_run` workflows run in the default branch context and don't appear as status checks on the PR page. Without explicit commit statuses, PR authors and reviewers have no visibility that evals are pending, waiting for approval, running, or complete. + +Status transitions: +- **pending** — set immediately when the discover job starts ("Eval run in progress...") +- **pending** — updated if the author is untrusted ("Waiting for approval in eval-protected environment") +- **success** — eval run completed, results posted as PR review +- **failure** — approval rejected, or eval run failed + +All statuses use the same `context: 'Eval PR Run'` so GitHub updates the existing status rather than creating duplicates. + ## Trust Model ### Trusted Authors (auto-approve) From 9f0a6ebfeb4602abe3b904345564c66127e03b1e Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 17:22:33 +0200 Subject: [PATCH 16/18] fix(ci): resolve stale pending status when discover fails Change report-status condition from needs.discover.result == 'success' to always() so it runs even when discover fails. Adds discover result to the status logic with a specific failure message. Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/eval-pr-run.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.github/workflows/eval-pr-run.yml b/.github/workflows/eval-pr-run.yml index 4bae79b4b..a4df18df7 100644 --- a/.github/workflows/eval-pr-run.yml +++ b/.github/workflows/eval-pr-run.yml @@ -268,18 +268,18 @@ jobs: report-status: name: Report Status needs: [discover, gate, run-evals] - if: >- - always() && - needs.discover.result == 'success' + if: always() runs-on: ubuntu-latest steps: - name: Set final commit status env: + DISCOVER_RESULT: ${{ needs.discover.result }} EVALS_RESULT: ${{ needs.run-evals.result }} GATE_RESULT: ${{ needs.gate.result }} uses: actions/github-script@v7 with: script: | + const discoverResult = process.env.DISCOVER_RESULT; const evalsResult = process.env.EVALS_RESULT; const gateResult = process.env.GATE_RESULT; @@ -287,6 +287,9 @@ jobs: if (evalsResult === 'success') { state = 'success'; description = 'Eval run completed — results posted as PR review'; + } else if (discoverResult !== 'success') { + state = 'failure'; + description = 'Eval run failed — could not resolve PR context'; } else if (gateResult === 'failure') { state = 'failure'; description = 'Approval was rejected in eval-protected environment'; From 461672fc302f03b646717ab5d98ae254e7141c1e Mon Sep 17 00:00:00 2001 From: mrizzi Date: Fri, 8 May 2026 17:32:02 +0200 Subject: [PATCH 17/18] docs(specs): update report-status to reflect always() condition Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md index 3768aef7a..2e96c4a84 100644 --- a/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md +++ b/docs/specs/2026-05-07-eval-pr-fork-dispatch-design.md @@ -208,7 +208,7 @@ run-evals: ### Job 4: report-status -Sets a final commit status on the PR's head SHA via the Commit Status API. Runs after all other jobs complete (`always()` + `needs.discover.result == 'success'`). +Sets a final commit status on the PR's head SHA via the Commit Status API. Runs unconditionally (`always()`) after all other jobs complete so a pending status is never left unresolved — including when the discover job itself fails. This is necessary because `workflow_run` workflows run in the default branch context and don't appear as status checks on the PR page. Without explicit commit statuses, PR authors and reviewers have no visibility that evals are pending, waiting for approval, running, or complete. @@ -216,7 +216,7 @@ Status transitions: - **pending** — set immediately when the discover job starts ("Eval run in progress...") - **pending** — updated if the author is untrusted ("Waiting for approval in eval-protected environment") - **success** — eval run completed, results posted as PR review -- **failure** — approval rejected, or eval run failed +- **failure** — discover failed (could not resolve PR context), approval rejected, or eval run failed All statuses use the same `context: 'Eval PR Run'` so GitHub updates the existing status rather than creating duplicates. From 37266be8597295fa3c35db7880c756944b75fd0c Mon Sep 17 00:00:00 2001 From: mrizzi Date: Mon, 11 May 2026 10:17:42 +0200 Subject: [PATCH 18/18] docs(plans): mark plan as superseded by evolved implementation Co-Authored-By: Claude Opus 4.6 (1M context) --- docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md b/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md index ec11cecd2..9e80bb879 100644 --- a/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md +++ b/docs/superpowers/plans/2026-05-07-eval-pr-fork-dispatch.md @@ -1,5 +1,7 @@ # Eval PR Fork Dispatch Implementation Plan +> **Superseded:** The implementation evolved past this plan during security hardening. The artifact was eliminated entirely — Stage 2 now derives all data from the GitHub API and git diff. See [the spec](../specs/2026-05-07-eval-pr-fork-dispatch-design.md) for the current design. + > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Split `eval-pr.yml` into a two-stage `workflow_run` dispatch pattern so fork PRs can run evals with secret access, gated by a collaborator trust check.