Skip to content

[#282] feat: coverage baseline + CI guardrail (tier-aware pipeline gate)#368

Merged
rucka merged 4 commits into
mainfrom
feat/282-coverage-baseline-ci-guardrail
Jul 24, 2026
Merged

[#282] feat: coverage baseline + CI guardrail (tier-aware pipeline gate)#368
rucka merged 4 commits into
mainfrom
feat/282-coverage-baseline-ci-guardrail

Conversation

@rucka

@rucka rucka commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

What Changed

An opt-in coverage baseline + regression guardrail, consumed as a job by the tier-aware pre-merge pipeline (#258) — never a second, parallel CI mechanism, and off by default (the twin of Pre-merge tiering):

  • Coverage guardrail flag (way-of-working: dataset template + pair adoption): disabled (default). A project opts in with enabled; absent the flag there is no coverage job and no tech/coverage-baseline.md.
  • Asset coverage-gate.sh (dataset + root mirror, provider-agnostic shell — ships to target-project pipelines like Tier-aware pre-merge pipeline — reads classification tags only, fail-safe red #258's tier-resolve.sh): coverage_gate <tier> <type> <current-%> <config-file> blocks a regression below the committed baseline at every tier; maintain/improve always passes. target_for_type / baseline_for_type resolve per-type config (backend/frontend/shared/default). Fail-safe on a missing coverage report: block at 🔴, warn at 🟢/🟡. Reads adoption config + a coverage number ONLY — no diff/code/path inspection (D18, grep-audited by the smoke test).
  • Persistence = human-committed (fix for the Major finding). The guardrail is live only once a human commits a baseline.<type>=NN line to the config. The gate never persists a baseline itself — a CI checkout is ephemeral, so a written baseline would be discarded and coverage could drift down run after run with the guard never firing. Bootstrapping is therefore advisory: on an unset/missing/corrupt baseline the gate prints the suggested baseline.<type>=NN to stderr and passes (bootstrap-only mode), without writing anything. Automated commit-back of a bootstrapped baseline is provider-specific and tracked separately — Coverage baseline auto commit-back — CI persists/ratchets the baseline (opt-in, on top of #282) #372. The misleading "machine-maintained" wording is removed.
  • Asset coverage-config-example.md (dataset + root): documents the config format (target.<type>, baseline.<type>, target.default, exclude) + KB-sensible defaults, the persistence model, and that exclude is applied by the adopter to their own coverage tool's config (the gate does not read it).
  • Guideline tier-aware-pipeline.md (dataset + root mirror): the coverage job is opt-in — emitted only when Coverage guardrail: enabled (documented like the tiering opt-in); a "Coverage guardrail (opt-in …)" section documents it's a job consumed by this pipeline, not a parallel mechanism.
  • setup-gates SKILL.md (dataset + mirror): a guided-setup question — "Enable the coverage guardrail? [default No]" (guided/quick convention, Guided/Quick Setup Convention — unify interactive vs default resolution across setup skills #276). enabled ⇒ writes the flag, creates tech/coverage-baseline.md from the KB template, and wires the coverage job; disabled/quick ⇒ nothing. Idempotent.
  • Docs site tag-driven-gates.mdx: "Coverage guardrail (opt-in, off by default)" section — opt-in, human-committed baseline, advisory bootstrap, adopter-applied exclude.
  • Removed .pair/adoption/tech/coverage-baseline.md (pair's own baseline) — it violated the default-absent principle; the example lives in the KB only.
  • Smoke test scripts/smoke-tests/scenarios/coverage-gate.sh (wired into run-all.sh): regression-blocks at every tier, maintain/improve passes, per-type target selection, advisory bootstrap (suggests on stderr, does NOT persist), corrupt-baseline advisory (no overwrite), CRLF baseline honored (regression test for the Minor fix), missing-report fail-safe by tier, D18 grep audit, opt-in + persistence + exclude documentation audits, and a no-"machine-maintained" guard — all PASS.

Why This Change

Closes story #282 of epic #209 (Code Quality & Testing foundations), epic AC3 / R7.3: test coverage should never silently drop, without demanding an absolute wall on a legacy codebase. Plugs into #258's pipeline as an opt-in gate (per epic #210's rule: no parallel CI mechanism, no gate re-deriving classification criteria of its own).

Design decisions

  • Opt-in, off by default. The guardrail is the twin of Pre-merge tiering: generated only on explicit Coverage guardrail: enabled. Adoption defaults (pair included) carry no coverage job and no baseline file.
  • Baseline is human-committed; bootstrap is advisory (persistence = A). The gate suggests, the human commits. This closes the review's Major finding: the guardrail is never silently inert because the docs no longer promise machine-maintained persistence, and the mechanism is explicitly "commit to activate." Automated commit-back ⇒ Coverage baseline auto commit-back — CI persists/ratchets the baseline (opt-in, on top of #282) #372.
  • Shipped shell asset, not a pair-cli TS module. coverage-gate.sh runs inside the target project's CI, exactly like Tier-aware pre-merge pipeline — reads classification tags only, fail-safe red #258's tier-resolve.sh — smoke-tested pipeline asset, per the tier-resolve precedent, not the gate-tooling ADL's "tested module not script" rule (which governs pair's OWN gates).

Testing

smoke coverage-gate.sh: PASS (incl. CRLF-honored, advisory-no-persist, opt-in/persistence/exclude doc audits, no "machine-maintained")
smoke tier-aware-gate.sh: PASS (no regression from the guideline edit)
@pair/knowledge-hub vitest + check:links: PASS (368 tests; dataset + root links valid)
mdlint (knowledge-hub + root adoption): PASS
docs:staleness: PASS (39 skills, 8 commands in sync)
Root mirror regenerated via `node apps/pair-cli/dist/cli.js update --offline --source packages/knowledge-hub/dataset`; coverage-gate.sh + coverage-config-example.md byte-identical dataset vs root.

Dogfood: coverage guardrail attivo sulla CI di pair

pair adotta la propria capability e la wira sulla CI reale — senza cambiare il default del framework (il template nel dataset resta disabled; opt-in solo a livello di progetto pair).

  • Opt-in pair. .pair/adoption/tech/way-of-working.md (Quality Gates): Coverage guardrail: enabled. Il dataset template resta disabled (verificato: packages/knowledge-hub/dataset/.pair/adoption/tech/way-of-working.md invariato).

  • Baseline = stato corrente misurato. Nuovo .pair/adoption/tech/coverage-baseline.md dal formato di coverage-config-example.md. Metrica: line coverage (total.lines.pct da istanbul coverage-summary.json). Solo i due package che emettono json-summary e girano già nella coverage-run CI (turbo test:coverage) alimentano il gate:

    Tipo Package Lines % misurata baseline target
    shared @pair/brand 85.04 85 90
    frontend @pair/website 20.16 20 25

    Baseline floored dall'attuale per assorbire il jitter float run-to-run: il gate passa sullo stato corrente e blocca solo una regressione reale (non unset). Gli altri package (knowledge-hub, content-ops, dev-tools, pair-cli) hanno test:coverage ma emettono solo text/html/lcov (no json-summary) e non sono nella coverage-run CI → non ancora nel gate (follow-up documentato nel file adoption).

  • Wiring CI (.github/workflows/ci.yml). Step standalone minimale Coverage guardrail (regression gate — dogfood #282), dopo lo step esistente Run coverage thresholds (brand, website) (mantenuto, coesiste). Lo step: source .pair/knowledge/assets/coverage-gate.sh → estrae total.lines.pct da ogni coverage-summary.json via node → chiama coverage_gate <tier> <type> <pct> .pair/adoption/tech/coverage-baseline.md. NON la pipeline tier-aware completa (Tier-aware pre-merge pipeline — reads classification tags only, fail-safe red #258). Tier: valore conservativo fisso red → un report di coverage mancante fallisce fail-safe (nessun pass silenzioso).

  • Sicurezza / non rompere le PR di pair. Step eseguito localmente sullo stato attuale ⇒ PASS (brand 85.04% ≥ 85, website 20.16% ≥ 20; entrambi advisory-warn sotto il target graduale, non bloccano). Verificato inoltre che il gate BLOCCA su regressione simulata (84.9% shared, 19.5% frontend) e passa su ==baseline. La source del gate non attiva l'entrypoint CLI quando eseguita come script file (come fa GitHub Actions run:), quindi nessun abort spurio sotto set -euo pipefail.

  • Scope. Toccati solo .pair/adoption/** di pair + .github/workflows/ci.yml. Nessuna modifica al default del dataset.

Dependencies & Related Work

Refs: #282

@github-actions
github-actions Bot temporarily deployed to Website Preview July 24, 2026 05:35 Inactive
@rucka

This comment has been minimized.

- coverage-gate.sh asset (dataset+root mirror): provider-agnostic baseline
  regression guardrail — blocks a drop below the established baseline at every
  tier, maintain/improve passes; per-type targets; baseline bootstrapping;
  fail-safe (block at red / warn lower) on a missing report. Reads adoption
  config + a number only, no classification criteria (D18).
- coverage-config-example.md asset (dataset+root): tech/coverage-baseline.md
  config format + KB-sensible defaults (backend 80 / frontend 60 / shared 90).
- tier-aware-pipeline.md (dataset+root): coverage job in the YAML + "Coverage
  guardrail" section (job consumed by this pipeline, not a parallel mechanism).
- adoption: pair's coverage-baseline.md (targets, baselines bootstrapped) +
  way-of-working Quality Gates bullet.
- docs site: tag-driven-gates.mdx "Coverage never silently drops" section.
- smoke test coverage-gate.sh wired into CI_TESTS (regression/improve/bootstrap/
  per-type/corrupt/missing-report + no-criteria + guideline audit — all PASS).

Refs: #282
@rucka
rucka force-pushed the feat/282-coverage-baseline-ci-guardrail branch from c638505 to 32a2e19 Compare July 24, 2026 06:44
@github-actions
github-actions Bot temporarily deployed to Website Preview July 24, 2026 06:44 Inactive
…tion)

Rework per maintainer decisions on PR #368:
- opt-in reframe: `Coverage guardrail: disabled` (default) flag, twin of
  Pre-merge tiering; coverage job generated only when enabled (wow template +
  pair adoption + tier-aware-pipeline + docs).
- persistence=A (fix Major): drop "machine-maintained"; bootstrap is advisory
  only (suggests baseline.<type> on stderr, never persists — CI checkout
  ephemeral); baseline is human-committed; commit-back deferred to #372.
- Minors: cov_config_value trims CR/whitespace (CRLF config honored);
  bootstrap advisory-only (no out-of-fence write); exclude is adopter-applied
  (not read by gate) — documented.
- remove pair's own tech/coverage-baseline.md (violated default-absent).
- setup-gates: guided "Enable the coverage guardrail? [default No]" (#276),
  enabled => flag + create coverage-baseline.md + wire job; idempotent.

Mirror regenerated via `pair update`; smoke coverage-gate.sh + tier-aware-gate.sh
+ knowledge-hub vitest/check:links + mdlint + docs:staleness all green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to Website Preview July 24, 2026 07:01 Inactive
@rucka

This comment has been minimized.

@rucka

This comment has been minimized.

@rucka

rucka commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

@/private/tmp/claude-501/-Users-gianlucacarucci5-projects-pair/6b978b2a-c3b5-4d25-be2b-01366429c4a9/scratchpad/c368cur.md

Adopt the story #282 capability on pair's own CI, without changing the
framework default (dataset template stays `disabled`).

- way-of-working: `Coverage guardrail: enabled` (pair opt-in only).
- tech/coverage-baseline.md: per-type targets + baselines measured on
  current state, floored to absorb jitter — shared(@pair/brand)=85
  (lines 85.04%), frontend(@pair/website)=20 (lines 20.16%).
- ci.yml: minimal standalone `Coverage guardrail` step after the existing
  coverage-thresholds step (kept). Sources coverage-gate.sh, reads
  total.lines.pct from each package's istanbul coverage-summary.json, calls
  coverage_gate at a conservative fixed tier=red (missing report fails safe).
  Blocks a regression below baseline; PASSES on current coverage.

Gate verified locally: PASS on current state; BLOCKS on simulated regression.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to Website Preview July 24, 2026 08:12 Inactive
0.04pp headroom (85.04→85) would false-block on trivial line churn/jitter.
Set baselines ~1pp below measured (shared 84, frontend 19) — still blocks a
genuine regression, absorbs run-to-run noise. Dogfood-safety on pair's own CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to Website Preview July 24, 2026 08:22 Inactive
@rucka
rucka merged commit 0628902 into main Jul 24, 2026
3 checks passed
@rucka
rucka deleted the feat/282-coverage-baseline-ci-guardrail branch July 24, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant