Skip to content

feat(ci): scitex-dev ci why — the ecosystem CI-failure-reading primitive (joins ci runner) - #345

Closed
ywatanabe1989 wants to merge 2 commits into
developfrom
feat/ecosystem-ci-why
Closed

feat(ci): scitex-dev ci why — the ecosystem CI-failure-reading primitive (joins ci runner)#345
ywatanabe1989 wants to merge 2 commits into
developfrom
feat/ecosystem-ci-why

Conversation

@ywatanabe1989

@ywatanabe1989 ywatanabe1989 commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

What

The ecosystem CI-failure-reading primitive + its thin CLI verb scitex-dev ci why.

Final CLI path is scitex-dev ci why (top-level ci group, joining ci runner) — not ecosystem ci why. scitex-dev drops the redundant middle dev (that word is already its package name); downstream consumers surface it as <cli> dev ci why. (Branch name feat/ecosystem-ci-why kept as-is to avoid churning the PR.)

Reading CI status is one word (failure); reading why has been tens of thousands of lines, so a bounded-context agent gets steered to the cheap word and the word replaces the reason. This inverts the price: fetch a failing run's --log-failed once, distil it to a few hundred bytes.

The primitive (SSOT) — scitex_dev.ci.why

Importable by ANY project (sac, the umbrella, …), which supply only a thin verb over it:

from scitex_dev.ci.why import explain_ci_run, render_text

for run in explain_ci_run("712"):        # PR#, run id, branch, or None
    print(render_text(run))
  • explain_ci_run(target, *, run_gh=None, repo=None) -> list[RunFailures] — a PR can front >1 failing run, hence a list.
  • explain_run(run_id, ...) -> RunFailures, resolve_run_ids(target, ...) -> list[str]
  • Parser helpers (pure, network-free): clean_log_line, split_log_by_job, parse_job_context, parse_failed_log — four tiers: pytest short test summary FAILED ids -> FAILURES block E lines -> ##[error] annotations -> tail fallback.
  • Injectable run_gh gh-seam (the only network touch) + CIWhyError (UNKNOWN is raised loudly, never swallowed into a reassuring "no failures").

The CLI verb

scitex-dev ci why <target> — the why verb attached to the existing top-level ci group, alongside ci runner (both coexist; runner is untouched). --json, -R/--repo. Tri-state exit: 0 green / 1 red (reason printed) / 2 could-not-read.

Watch it fail (real live runs)

scitex-dev run 29312551895 (workflow tests) — pytest tier, all 3 matrix legs:

pytest-matrix-on-ubuntu-py3.11 (py3.11, ubuntu)
  FAILED tests/develop/test_audit.py::test_audit_all_clean - AssertionError: audit-all reported violations for 'scitex-dev' (exit=1).
  E           AssertionError: audit-all reported violations for 'scitex-dev' (exit=1).
  -> https://github.com/scitex-ai/scitex-dev/actions/runs/29312551895/job/87019303421

sac run 29446283736 (workflow quality, a setup failure with no pytest) — ##[error] tier:

scitex-dev-quality-audit-on-ubuntu-latest (ubuntu-latest)
  ##[error] error: could not lock config file .../.gitconfig: File exists
  ##[error] fatal: no submodule mapping found in .gitmodules for path '.tmp-audit/scitex-browser'
  ##[error] Unknown system error -116: ... copyfile ... uv

Both extract the reason, not the whole log.

Tests — no mocks

  • Parser off real-shaped GitHub-Actions log STRINGS (pytest-failure log + a ##[error] setup-failure log).
  • Resolver + explain_* through an injected run_gh callable (offline).
  • CLI end-to-end against a real fake gh on PATH (a real subprocess; PATH restored, no monkeypatch — PA-306). A coexistence test asserts ci runner and ci why both resolve.

Full slice green (156 passed — the new primitive/CLI tests plus the pre-existing ci/runner suite and the ecosystem registry/categories regression).

Notes

…re-reading primitive

Reading CI status is one word (`failure`); reading why has been tens of
thousands of lines. Add the reusable SSOT primitive that inverts that
price — it fetches a failing run's log once and distils it to a few
hundred bytes (failing test ids / assertion lines / a setup `##[error]`)
— plus the thin CLI verb any consumer's own verb can mirror.

Primitive scitex_dev.ci.why (importable by any project):
  explain_ci_run(target, *, run_gh=None, repo=None) -> list[RunFailures]
  + parser helpers clean_log_line / split_log_by_job / parse_job_context
    / parse_failed_log (four-tier: pytest summary -> FAILURES E-lines ->
    ##[error] annotations -> tail fallback)
  + the injectable run_gh gh-seam and CIWhyError (UNKNOWN never reads as
    green). Everything but the gh-seam is pure/string-based.

CLI: scitex-dev ecosystem ci why <target> (a new ci subgroup under
ecosystem) resolves a PR# / run id / branch / nothing to the failing
run(s); --json and -R/--repo; tri-state exit (0 green / 1 red / 2
could-not-read).

Lifted generically from sac's shipped _ci_why.py (no sac specifics).
Tests carry no mocks: the parser runs off real-shaped GitHub-Actions log
strings, the resolver through an injected run_gh callable, and the CLI
end-to-end against a real fake gh on PATH.
…ci runner)

Per operator ruling the CLI verb lives on the EXISTING top-level `ci`
group (which already holds `ci runner`), not under `ecosystem`. scitex-dev
drops the redundant middle `dev` because "dev" is already its package
name — consumers use `<cli> dev ci why`, but scitex-dev itself is just
`scitex-dev ci why`.

The library primitive scitex_dev.ci.why is UNCHANGED. Only the verb
wiring moved:
  - new scitex_dev/ci/_why_cli.py::register_ci_why_command attaches `why`
    to the ci group in _root.py (which now captures the group returned by
    register_ci_runner_commands, then registers `why` beside `runner`);
  - the `ecosystem ci` subgroup + its Discovery category entry are
    reverted;
  - the ci group docstring lists the new `why` verb.

Tests updated: invoke `ci why` (not `ecosystem ci why`) and assert
`ci runner` + `ci why` coexist. Everything else (four-tier parser, run_gh
seam, CIWhyError, --json, tri-state exit, no-mocks) unchanged.
@ywatanabe1989 ywatanabe1989 changed the title feat(ecosystem): scitex-dev ecosystem ci why — the ecosystem CI-failure-reading primitive feat(ci): scitex-dev ci why — the ecosystem CI-failure-reading primitive (joins ci runner) Jul 16, 2026
@ywatanabe1989

Copy link
Copy Markdown
Collaborator Author

Reviewed — sound, merge once develop is green. (GitHub won't let me formally approve: the whole fleet pushes under one account, so it reads as my own PR. Recording the review here instead.)

WHAT I VERIFIED rather than took from the description:

  1. The tier-4 fallback in _parse.py behaves correctly — when nothing structured is found (not (fail.failed_tests or fail.assertions or fail.errors)) it falls back to the cleaned tail rather than returning an empty result. An unparseable log yields raw evidence, never a reassuring "no failures". That is the single most important property for this tool.

  2. Tri-state exit (0 green / 1 red / 2 could-not-read) keeps "couldn't read" distinct from "nothing wrong", and CIWhyError is raised rather than swallowed. An UNKNOWN cannot render as a success pole.

Those two are the difference between a CI-reading tool and a CI-reassurance tool, and this gets them right.

WHY IT MATTERS MORE THAN WHEN IT WAS WRITTEN: today I merged a PR that broke develop, then merged four more on top of the red for an hour without noticing — because I had convinced myself the runners were dead and stopped reading the gate at all. This primitive is the direct countermeasure: it makes the reason as cheap as the status. It also implicitly proved itself during that incident, since the only reason I could bisect the break was that the log named the failing assertion.

Blocked only on develop going green, which is my own breakage (#355 §1/§4 violations, #359 orphaned tests, #354 an ungenerated import gate). Three fix PRs in flight; this one is unrelated and not a cause.

Sorry it sat open — that's on me, not on the work, which was ready and had already been reworked twice to my naming rulings.

@ywatanabe1989

Copy link
Copy Markdown
Collaborator Author

Closing under the 3-day PR rule (operator: 「PRの賞味期限は3日にしましょう」「3日より古いものは強制close」「腐ってる、新しく作った方が早い」).

Opened 2026-07-16, now 3 days old and DIRTY — develop has moved substantially underneath it (audit-violation fix, versioning fastpath, PS-142 truncation fix, hosted-runner migration all landed since). gh pr update-branch refuses: real conflicts, not a stale base.

NOT ABANDONED — the intent is recorded and will be rebuilt fresh off current develop:
scitex-dev ci why — the ecosystem CI-failure-reading primitive. Parse a failed run into the actual failing tests/assertions rather than making every agent hand-roll log greps. Tracked on card scitex-dev-closed-stale-pr-intents-rebuild-fresh-20260718.

Rebuilding costs less than resolving a 3-day-old conflict set, and the result is based on code that actually exists now. That is the whole point of the rule.

@github-actions github-actions Bot locked and limited conversation to collaborators Jul 18, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant