Skip to content

[QUESTION] PR merge-gate rule requires green gh pr checks, but this repo's CI never attaches to PRs #154

Description

@bermudas

Found while working #127 (ELITEA-1739 automation, PR #130)

The situation

PR #130 (ELITEA-1739 "Search Skills by Name" automation) is fully ready:

  • Implementer: green 3/3 local runs.
  • A fresh, independent reviewer approved it after triangulating the original test case, the automation spec, and the code, and re-running it live twice.
  • I additionally triggered this repo's CI workflow (Playwright Tests, run 29105930248) against the PR's branch to get an independent, full-suite confirmation. Downloaded the JSON test report: the new test (ELITEA-1739: search skills by name) passed. The run's overall result is "failure," but that's from 12 pre-existing, already-documented failures in unrelated modules (agents/pipelines/modal-handling/lazy-loading/artifacts — e.g. the known GH#29 defect) — not from anything in this PR.

The actual blocker

My working rule for this task is: merge only on a green gh pr checks exit — never on my own reading of test output. But gh pr checks 130 reports no checks at all, and always will under the current CI config: .github/workflows/playwright.yml is deliberately workflow_dispatch-only (manual trigger, no pull_request trigger — see the workflow file's own header comment, and .agents/testing.md § CI integration, which says not to add a pull_request trigger "without a separate, explicit decision"). Manually running the workflow against the PR's branch (as I did) produces a run, but GitHub does not attach it to the PR as a check — gh pr checks still reports nothing, regardless.

So as currently configured, no PR on this repo can ever satisfy "green gh pr checks" — not because anything is broken, but because there's structurally no PR-attached check to be green. This isn't something I can safely decide unilaterally: adding a pull_request trigger is exactly the "separate, explicit decision" the workflow file's own comment says to wait for, and it's out of scope for the ELITEA-1739 case I was dispatched to automate.

Options I see

  1. Add a pull_request trigger to .github/workflows/playwright.yml (framework-scale CI change) so future PRs get a real, PR-attached check. Tradeoff: every PR would then run the entire suite (currently ~43 minutes, workers:1) including the 12 known-defect reds, so gh pr checks would likely never go fully green anyway unless those known-defects are marked in a way CI treats as non-blocking (e.g. a required-checks allowlist, or splitting the workflow to run only changed-module tests).
  2. Redefine the merge-gate mechanic for this repo specifically: since a PR-attached check isn't available, treat "reviewer APPROVED + a manually-triggered full-suite run showing the new test passing, with no new failures beyond the existing known-defect baseline" as the accepted merge evidence for unattended runs — i.e., amend this project's merge-gate rule/seed to match what's actually achievable here, rather than a rule written for a repo with PR-triggered CI.
  3. Something else a human sees that I don't.

My recommendation

Option 2, scoped narrowly: treat a manually-triggered workflow_dispatch run (like the one I already have for PR #130) as valid merge evidence specifically when (a) the new/changed test(s) pass, and (b) the only failures present already match this repo's documented known-defect baseline (.agents/testing.md § Known issues / § CI integration's open infra-crash bucket) with no new failures introduced. Option 1 is a legitimate future improvement but a bigger, separate framework-scale decision (would need a "known-defects don't block" mechanism to avoid making PRs permanently red by design).

Parking #127 as Blocked pending this decision — PR #130 itself is otherwise done (implementer green, reviewer approved, manually-verified CI pass on the specific test).

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions