diff --git a/.claude/skills/pair-capability-setup-gates/SKILL.md b/.claude/skills/pair-capability-setup-gates/SKILL.md index f8c75e23..9079ac1a 100644 --- a/.claude/skills/pair-capability-setup-gates/SKILL.md +++ b/.claude/skills/pair-capability-setup-gates/SKILL.md @@ -50,7 +50,7 @@ Configure CI/CD quality gates for the project. Reads quality assurance guideline 1. **Act**: Read quality assurance guidelines: - [quality-assurance.md](../../../.pair/knowledge/guidelines/technical-standards/git-workflow/quality-assurance.md) — gate types and checklists - [quality-gates.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-standards/quality-gates.md) — gate framework and registry format - - [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md) — the **opt-in** optimization that reduces checks per risk tier (tags only, fail-safe red); default is the full suite on every PR, tier-aware only when the project opts in + - [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md) — the **opt-in** optimization that reduces checks per risk tier (tags only, fail-safe red); default is the full suite on every PR, tier-aware only when the project opts in. Also defines the **opt-in coverage guardrail** job (off by default) and its [config format](../../../.pair/knowledge/assets/coverage-config-example.md) - [quality-model.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-model.md) §4 — the single source of the tier→checks matrix the pipeline projects (never re-declared in generated config) - [shared-config-packages.md](../../../.pair/knowledge/guidelines/code-design/quality-standards/shared-config-packages.md) — shared-config-package pattern, per-type overrides, `tools/*` reference implementation - [secret-scanning.md](../../../.pair/knowledge/guidelines/quality-assurance/security/secret-scanning.md) — the deterministic, required-at-every-tier CI job this skill provisions (R6.5, D24) — never a skill, never an LLM judgment call @@ -98,7 +98,16 @@ Configure CI/CD quality gates for the project. Reads quality assurance guideline - **Quick mode / non-interactive / CI** — take the default: **No** (full pipeline). Never prompt. - Record the resolved choice for Step 4: `TIERING = enabled | disabled` (default `disabled`). -4. **Verify**: Developer approves gate configuration and the tiering choice is resolved (`disabled` unless explicitly opted in). +4. **Act — coverage guardrail opt-in** (same [guided/quick convention](../../../.pair/knowledge/guidelines/technical-standards/ai-development/skill-conventions/guided-quick-setup.md), default **No** — the twin of the tiering opt-in above): the coverage regression guardrail is off by default; a project opts in like any other input. + - **Guided mode** (interactive/TTY) — ask **one** question, pre-filled with the default: + + > Enable the coverage guardrail? This adds a job that blocks a PR whose test coverage drops **below a committed baseline** (maintaining/improving passes — it is not an absolute wall). Off by default; the baseline is human-committed and bootstraps advisory-only on first run. **[default: No]** + + Default (Enter) ⇒ **No** (no coverage job). Only an explicit "Yes" opts in. + - **Quick mode / non-interactive / CI** — take the default: **No**. Never prompt. + - Record the resolved choice for Step 4: `COVERAGE_GUARDRAIL = enabled | disabled` (default `disabled`). + +5. **Verify**: Developer approves gate configuration; the tiering choice (`disabled` unless opted in) and the coverage-guardrail choice (`disabled` unless opted in) are both resolved. ### Step 4: Write Configuration @@ -106,13 +115,15 @@ Configure CI/CD quality gates for the project. Reads quality assurance guideline - Set quality gate command (e.g., `pnpm quality-gate`) - Write or update Custom Gate Registry table - **Pre-merge tiering flag**: if `TIERING = enabled` (Step 3.3 opt-in), record `Pre-merge tiering: enabled` in the Quality Gates section. If `disabled` (the default), either omit the line or record `Pre-merge tiering: disabled` — both mean full-suite. Never write `enabled` without the explicit opt-in. + - **Coverage guardrail flag**: if `COVERAGE_GUARDRAIL = enabled` (Step 3.4 opt-in), record `Coverage guardrail: enabled` in the Quality Gates section; if `disabled` (the default), either omit the line or record `Coverage guardrail: disabled`. Never write `enabled` without the explicit opt-in. **When enabled**, also create `tech/coverage-baseline.md` from the KB template ([coverage-config-example.md](../../../.pair/knowledge/assets/coverage-config-example.md)) with the per-type targets for the adopted stack and the baselines left unset (bootstrap-only until a human commits them — see that example's Persistence note). **Idempotent**: if the flag is already `enabled` and `tech/coverage-baseline.md` already exists, confirm rather than overwrite. 2. **Act**: Generate the **pre-merge pipeline** appropriate for the adopted stack and hosting. **Which pipeline depends on `TIERING`:** - GitHub Actions → `.github/workflows/quality.yml` - GitLab → `.gitlab-ci.yml` quality stage - Other → document commands for manual pipeline setup - **`TIERING = disabled` (default) → full-suite pipeline**: every PR runs base (install + lint + type + build) **and** unit **and** integration/E2E **and** coverage — all jobs unconditional, no tag resolution, no `risk:*` reads. This is the safe default; do **not** generate the tier-aware jobs. - **`TIERING = enabled` (opt-in) → tier-aware pipeline** following [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md): the pipeline **reads the PR's `risk:*` tag only** (via [`tier-resolve.sh`](../../../.pair/knowledge/assets/tier-resolve.sh) — tags only, no classification criteria of its own, D18) and schedules the [quality-model.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-model.md) §4 matrix: base every tier, unit from 🟡, integration/E2E from 🔴; **untagged/unknown/malformed ⇒ 🔴** (fail-safe); a required suite missing at a tier ⇒ **explicit failure** (`require_suite`), never a silent pass. Do not copy any tier threshold into the generated config — it is a projection of the model, not a second source. - - **Wire required checks**: mark `base` and `secret-scan` as required status checks on the protected branch. In full-suite mode all suite jobs are required; in tier-aware mode the tier-scoped jobs are required-when-scheduled (or via one aggregating gate job). A red gate blocks merge either way. + - **Coverage job (opt-in)**: emit the `coverage` job from [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) **only when `COVERAGE_GUARDRAIL = enabled`** (Step 3.4). It sources [`coverage-gate.sh`](../../../.pair/knowledge/assets/coverage-gate.sh) (config + a number only, no criteria — D18), reads `tech/coverage-baseline.md`, and blocks a regression below the committed baseline (maintain/improve passes; bootstrap-only when unset). With the flag absent (default) do **not** emit a coverage job in either pipeline mode. + - **Wire required checks**: mark `base` and `secret-scan` as required status checks on the protected branch. In full-suite mode all suite jobs are required; in tier-aware mode the tier-scoped jobs are required-when-scheduled (or via one aggregating gate job). A red gate blocks merge either way. The coverage job (when enabled) is required-when-scheduled like the other tier-scoped jobs. - **Post-merge staging** (on merge to the main branch): **build + deploy only, no gate re-run** — the gate already passed pre-merge (identical in both modes). 3. **Act**: Write the secret-scanning job — required, not an optional proposal line, in **both** pipeline modes: resolve the scanner (Argument > Adoption's Custom Gate Registry override > KB default gitleaks), then write the [CI Job Template](../../../.pair/knowledge/guidelines/quality-assurance/security/secret-scanning.md#ci-job-template-github-actions) into the same pipeline file as a `required` job, and provision a starting `.gitleaks.toml` at the project root (see the [allowlist mechanism](../../../.pair/knowledge/guidelines/quality-assurance/security/secret-scanning.md#allowlist-mechanism-adoption-controlled)) if one doesn't already exist. Never write the job with `continue-on-error` — fail-closed is not optional (R6.5). 4. **Verify**: Configuration files written, including the pre-merge pipeline (full-suite by default; tag-only tier-aware resolution + tier-scoped jobs only when `TIERING = enabled`) + required-check wiring, the post-merge staging (build+deploy only) config, the secret-scanning job, and `.gitleaks.toml`. If tiering is enabled, grep the generated pipeline for classification tokens (schema/diff/path heuristics) — there must be none (D18). If disabled, confirm no `risk:*` resolution was generated and every suite job runs unconditionally. @@ -152,6 +163,7 @@ GATE CONFIGURATION COMPLETE: ├── Pre-production: [N gates configured | N/A] ├── Pipeline: [file path | manual — full-suite pre-merge (default: every PR runs all suites) OR tier-aware pre-merge (opt-in: tags only, fail-safe 🔴) + post-merge staging build+deploy] ├── Pre-merge tiering: [disabled (default — full suite every PR) | enabled (risk-tier-scoped, opt-in)] +├── Coverage guardrail: [disabled (default — no coverage job) | enabled (opt-in — coverage job + tech/coverage-baseline.md created)] ├── Required checks: [base + secret-scan required; full-suite: all suite jobs required | tier-aware: tier-scoped jobs required-when-scheduled] ├── Secret Scan: [gitleaks (KB default) | — job + .gitleaks.toml written | already configured] ├── Shared configs: [package list | N/A — non-JS, documented pointer only] @@ -184,6 +196,7 @@ See [graceful degradation](../../../.pair/knowledge/guidelines/technical-standar - This skill **modifies files** — it writes to way-of-working.md, creates/updates CI/CD pipeline configuration (the pre-merge pipeline — full-suite by default, tier-aware only on opt-in — + post-merge staging + the required secret-scanning job and `.gitleaks.toml`), and provisions shared lint/format config packages + hook manager files (`.husky/` by default). - **Full checks are the default; risk-tier reduction is opt-in** (ADL 2026-07-20): unless the project sets `Pre-merge tiering: enabled` in way-of-working.md (Step 3.3 opt-in), the generated pipeline runs the **full suite on every PR** — the safe, pre-tiering behavior. Tiering only *reduces* checks on lower-risk PRs, so it must be enabled explicitly. +- **Coverage guardrail is opt-in, off by default** (story #282): the `coverage` regression job and `tech/coverage-baseline.md` are generated **only** when the project sets `Coverage guardrail: enabled` (Step 3.4 opt-in) — the twin of the tiering opt-in. Absent the flag, no coverage job and no baseline file. The baseline is human-committed; the gate bootstraps advisory-only (suggests on stderr, never persists — a CI checkout is ephemeral; automated commit-back is story #372). See [coverage-config-example.md](../../../.pair/knowledge/assets/coverage-config-example.md). - **When tiering is enabled, the generated pipeline reads classification tags only, never classifies** (D18): tier criteria live in [quality-model.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-model.md) §3/§4; the pipeline is the deterministic Automation layer that projects the matrix from the PR's `risk:*` tag. Untagged ⇒ 🔴 (fail-safe). See [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md). - **Secret scanning is CI config, not a judgment call** (D24, anti-complexity): this skill provisions the job mechanically; it never evaluates whether a diff contains a secret itself — that is gitleaks' (or the adopted scanner's) job at runtime, with no LLM in the loop. `/pair-capability-assess-security` never re-implements this — see that skill's own Notes. - **Idempotent** — see [idempotency convention](../../../.pair/knowledge/guidelines/technical-standards/ai-development/skill-conventions/idempotency.md). This skill's check: an already-configured project (incl. provisioned shared configs and hooks) is confirmed; update only on explicit developer request. Conflicting local config is always resolved by asking first (see Edge Cases above). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 17249063..7ea4334b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -95,5 +95,26 @@ jobs: run: pnpm test - name: Run coverage thresholds (brand, website) run: pnpm turbo test:coverage --filter=@pair/brand --filter=@pair/website + # Dogfood of the story #282 coverage guardrail on pair's own CI (opt-in: + # way-of-working `Coverage guardrail: enabled`). Standalone/minimal — NOT + # the full tier-aware pipeline (#258). Reuses the coverage run above: reads + # the line % from each package's istanbul coverage-summary.json and blocks + # a regression below the human-committed baseline in tech/coverage-baseline.md. + # Conservative fixed tier=red => a missing coverage report fails safe. + - name: Coverage guardrail (regression gate — dogfood #282) + run: | + set -euo pipefail + # shellcheck disable=SC1091 + source .pair/knowledge/assets/coverage-gate.sh + TIER=red + CFG=.pair/adoption/tech/coverage-baseline.md + pct() { + node -e "const fs=require('fs');process.stdout.write(String(JSON.parse(fs.readFileSync('$1','utf8')).total.lines.pct))" + } + BRAND_PCT="$(pct packages/brand/coverage/coverage-summary.json || true)" + WEBSITE_PCT="$(pct apps/website/coverage/coverage-summary.json || true)" + echo "coverage: brand(shared)=${BRAND_PCT}% website(frontend)=${WEBSITE_PCT}%" + coverage_gate "$TIER" shared "$BRAND_PCT" "$CFG" + coverage_gate "$TIER" frontend "$WEBSITE_PCT" "$CFG" - name: Run E2E tests run: pnpm --filter @pair/website e2e diff --git a/.pair/adoption/tech/coverage-baseline.md b/.pair/adoption/tech/coverage-baseline.md new file mode 100644 index 00000000..f912ace9 --- /dev/null +++ b/.pair/adoption/tech/coverage-baseline.md @@ -0,0 +1,49 @@ +# `tech/coverage-baseline.md` — pair coverage guardrail config + +pair's own coverage guardrail config (dogfood of story #282). This file exists because pair opted into the guardrail (`Coverage guardrail: enabled` in [way-of-working.md](./way-of-working.md)). The [`Coverage guardrail` step](../../../.github/workflows/ci.yml) reads this file with the shipped [`coverage-gate.sh`](../../knowledge/assets/coverage-gate.sh); it blocks a PR whose line coverage drops below a `baseline.` here, at every tier — maintaining or improving always passes. + +Format documented in the KB [config example](../../knowledge/assets/coverage-config-example.md): a single `key=value` block; the surrounding markdown fence/headings are ignored (lines matched with a plain `^key=`). + +## Types measured + +Only the two packages that emit an istanbul `coverage-summary.json` (`reporter: ['json-summary', …]`) and already run in CI via `turbo test:coverage` feed the gate. The type is the touched package's nature: + +| Package | Type | Nature | +| --------------- | ---------- | ---------------------------------------------------------- | +| `@pair/website` | `frontend` | Next.js app — UI validated primarily by Playwright e2e. | +| `@pair/brand` | `shared` | Reusable design-system component library, consumed widely. | + +The other packages (`knowledge-hub`, `content-ops`, `dev-tools`, `pair-cli`) have `test:coverage` but emit only `text`/`html`/`lcov` (no `json-summary`) and are not run in the CI coverage step, so they are not fed to the guardrail yet. Wiring them in is a follow-up (extend the CI coverage run to include them + add `json-summary` to their vitest config). + +## Baseline (measured current state) + +The metric is **line coverage** (`total.lines.pct` from `coverage-summary.json`). Baselines below were **measured on the current `main`/branch state** and set **~1pp below** the measured value, to absorb per-run float jitter and legitimate line churn (so the gate passes today and blocks only a genuine regression, not run-to-run noise): + +| Type | Measured lines % | `baseline` (~1pp margin) | `target` (gradual goal) | +| ---------- | ---------------- | ---------- | ----------------------- | +| `shared` | 85.04 (`@pair/brand`) | 84 | 90 | +| `frontend` | 20.16 (`@pair/website`) | 19 | 25 | + +`target.frontend` is deliberately modest: the website is e2e-covered by design (see the website vitest ADL 2026-07-12), so unit line coverage is intentionally low; below-target-but-above-baseline only warns. + +## Config + +```ini +# pair coverage guardrail — per-type gradual targets + committed baselines. +# Metric: line coverage (total.lines.pct from istanbul coverage-summary.json). +target.default=70 +target.shared=90 +target.frontend=25 + +# Committed baselines — the guardrail blocks a drop below these (every tier). +# Set ~1pp below the measured current state to absorb float jitter + legitimate +# line churn (blocks only a genuine regression, not run-to-run noise). +baseline.shared=84 +baseline.frontend=19 +``` + +## Notes + +- **Baseline-relative, not an absolute wall.** A drop below `baseline.` blocks; below the gradual `target` but at/above baseline only warns (R7.3: incremental improvement). +- **Human-committed.** The gate never persists a baseline itself; raise a baseline here by committing a new value once coverage improves. +- **Fail-safe.** If a coverage report is missing, the CI step runs the gate at a conservative fixed `red` tier, so a missing report blocks rather than silently passes. diff --git a/.pair/adoption/tech/way-of-working.md b/.pair/adoption/tech/way-of-working.md index 01fbfbce..09da2027 100644 --- a/.pair/adoption/tech/way-of-working.md +++ b/.pair/adoption/tech/way-of-working.md @@ -37,6 +37,7 @@ Maps this project's GitHub Projects board columns to the 5 canonical macrostates - `pnpm quality-gate` is the adopted project-level quality gate command. - Quality gate includes: type checking (`ts:check`), testing (`test`), linting (`lint`), formatting (`prettier:fix`), markdown lint (`mdlint:fix`). - **Pre-merge tiering**: `disabled` (default) — every PR runs the full pre-merge check suite. Set to `enabled` to opt into risk-tier-scoped pre-merge checks (lighter checks on lower-risk PRs) per [tier-aware-pipeline.md](../../knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md); `/pair-capability-setup-gates` reads this flag before generating the pipeline. +- **Coverage guardrail**: `enabled` — pair dogfoods its own capability: the [`Coverage guardrail` step](../../../.github/workflows/ci.yml) in CI sources [`coverage-gate.sh`](../../knowledge/assets/coverage-gate.sh), extracts the line-coverage % from each package's istanbul `coverage-summary.json`, and blocks a PR whose coverage drops below the human-committed baseline in [`tech/coverage-baseline.md`](./coverage-baseline.md) (maintaining/improving passes — not an absolute wall). The framework **default remains `disabled`** (the dataset template ships off); this line is pair's project-level opt-in only. See [coverage guardrail](../../knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) + [config format](../../knowledge/assets/coverage-config-example.md); `/pair-capability-setup-gates` reads this flag before generating the pipeline. - **Gate & tooling code:** a gate's logic lives in a tested module in its owning package (white-box unit tests); scripts/CLIs are thin entrypoints and a root gate delegates (`pnpm --filter `). Scripts are never unit-tested — CLI-level checks go to smoke tests. See ADL [2026-07-13-gate-tooling-code-in-tested-modules.md](../decision-log/2026-07-13-gate-tooling-code-in-tested-modules.md). Gate/tooling packages are organized by bounded context, not one package per tool family — a new tool family sharing an existing package's bounded context is a new folder there, not a new package. See [ADR-014](adr/adr-014-tool-package-boundary-by-bounded-context.md). - **Conformance tests** (`packages/knowledge-hub/src/conformance/`): one test file per target KB artifact (a `SKILL.md`, guideline, or template), not per introducing story — a new story extends the matching file's `describe` block instead of adding a new story-named file. See ADL [2026-07-18-conformance-test-per-file-not-per-story.md](../decision-log/2026-07-18-conformance-test-per-file-not-per-story.md). - **Monorepo tooling gotchas** (e.g. `pnpm --filter` bypassing turbo's `dependsOn` graph on a fresh checkout): documented once, centrally, in `DEVELOPMENT.md`'s `Turbo Caching` section — affected packages' READMEs carry only a short pointer, not a full copy. See ADL [2026-07-18-workspace-gotcha-doc-placement.md](../decision-log/2026-07-18-workspace-gotcha-doc-placement.md). diff --git a/.pair/knowledge/assets/coverage-config-example.md b/.pair/knowledge/assets/coverage-config-example.md new file mode 100644 index 00000000..b7d8c390 --- /dev/null +++ b/.pair/knowledge/assets/coverage-config-example.md @@ -0,0 +1,59 @@ +# `tech/coverage-baseline.md` — Example + +Illustrative coverage config for a project's `tech/coverage-baseline.md` adoption file. This config exists **only when a project opts into the coverage guardrail** (`Coverage guardrail: enabled` in `way-of-working.md`, via `/pair-capability-setup-gates`) — the guardrail is off by default, so a project without it has no such file. Copy the fenced block into your own adoption file and adjust the numbers — this example is committed only as a reference. The [coverage guardrail](../guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) reads this config with the shipped [`coverage-gate.sh`](coverage-gate.sh) helper. + +## What lives here + +Two kinds of value, in one parseable `key=value` block (order-independent; lines are read with a plain `^key=` match, so the surrounding markdown fence and headings are ignored): + +- **`target.`** — the *gradual* per-type coverage goal (human-set). Below the target but not below the baseline only **warns** — it is not a hard wall (R7.3: incremental improvement, not "must hit X% on day one"). +- **`baseline.`** — the *committed* coverage the guardrail protects (**human-committed** — see [Persistence](#persistence-the-baseline-is-human-committed) below). A drop below the baseline **blocks the merge**, at every tier. Leave it unset and the gate runs in **bootstrap-only mode**: it prints the current coverage as a suggested `baseline.=NN` to stderr and passes without blocking, until a human commits that line. +- **`target.default`** — the fallback target for any type without its own `target.` line. +- **`exclude`** — glob(s) for genuinely untestable surface (generated files, config). This is **applied by the adopter to their own coverage tool's config** — the gate does **not** read it and the pipeline snippet does **not** pass it. It is recorded here as the single place the exclusion intent is documented; wiring it into the coverage tool (e.g. istanbul `exclude`, vitest `coverage.exclude`) is the adopter's step. + +The `` is the touched code's type — `backend`, `frontend`, `shared`, or whatever your stack distinguishes; the pipeline passes the type matching the code under test so per-type targets apply (AC5). + +## Persistence: the baseline is human-committed + +The guardrail is **live only once a human commits a `baseline.=NN` line** to this file. The gate never persists a baseline itself: a CI checkout is ephemeral, so anything it wrote would be discarded when the runner is torn down — and coverage could then drift down run after run, each run re-suggesting a new lower baseline it never keeps, with the regression guard never firing. + +So bootstrapping is **advisory**: on an unset/missing/corrupt baseline the gate prints the suggested `baseline.=NN` to stderr and **passes** (bootstrap-only mode). Read the CI log, copy the suggested line into this committed config, and the guardrail becomes live on the next run. Automated commit-back of a bootstrapped baseline is provider-specific and tracked separately — see story **#372**. + +## KB-sensible defaults + +The guardrail is baseline-relative, so absolute targets are advisory. These are sensible starting goals; tune per project: + +| Type | Target | Rationale | +| ---------- | ------ | ------------------------------------------------------ | +| `backend` | 80 | Business logic / services — the highest test priority. | +| `frontend` | 60 | UI — some surface is exercised by E2E, not unit. | +| `shared` | 90 | Reused libraries — a regression here blasts widely. | +| `default` | 70 | Anything without a more specific type. | + +## Example + +```ini +# Coverage config — per-type gradual targets (human-set) + committed baseline. +# Leave a baseline. unset to run in bootstrap-only mode (the gate suggests +# a value on stderr and passes; commit it to activate the guardrail). +target.default=70 +target.backend=80 +target.frontend=60 +target.shared=90 + +# Glob(s) excluded from measurement — applied by the adopter to their coverage +# tool's own config (the gate does not read this key; it is documented here only). +exclude=**/*.generated.ts,**/*.config.ts + +# Committed baseline — the guardrail blocks a drop below these. Human-committed: +# unset/empty => bootstrap-only mode (the gate suggests a value on stderr, passes). +baseline.backend=82.5 +baseline.frontend=61 +# baseline.shared= (unset => bootstrap-only until a human commits a value) +``` + +## Notes + +- **No coverage tool is mandated.** The gate consumes a coverage percentage the pipeline extracts from whatever the adopted test framework emits (istanbul `coverage-summary.json`, LCOV, Cobertura, …). See the guideline's [coverage guardrail](../guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) section for the extraction snippet per stack. +- **Missing/corrupt baseline** runs in bootstrap-only mode: it suggests a value on stderr and passes with a warning — it never blocks everything, and it never persists a value on its own. +- **No coverage measured** (test tooling emitted nothing / tests did not run) fails safe: it blocks at the red tier and warns at lower tiers, never a silent pass. diff --git a/.pair/knowledge/assets/coverage-gate.sh b/.pair/knowledge/assets/coverage-gate.sh new file mode 100644 index 00000000..e2bb5bb6 --- /dev/null +++ b/.pair/knowledge/assets/coverage-gate.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +# coverage-gate.sh — provider-agnostic coverage baseline + regression guardrail +# for the tag-driven pre-merge gate (story #282). +# +# THIS FILE IS A GUARDRAIL, NOT A CLASSIFIER. It reads two inputs only: +# 1. the project's coverage config in adoption (per-type targets + a +# human-committed baseline) — see coverage-config-example.md; and +# 2. a coverage percentage the pipeline already extracted from whatever report +# the adopted test tooling produced (e.g. istanbul coverage-summary.json). +# It contains NO classification criteria (D18): it never inspects the diff, the +# code, file paths, or change size. The risk tier is decided upstream by +# `classify` and passed in only to choose the fail-safe behavior when NO coverage +# was measured (block at red, warn at lower tiers). +# +# Policy (story #282): +# - The guardrail blocks a REGRESSION below the committed baseline, at every +# tier — not "must hit X% absolute". Maintaining or improving passes (AC1/AC2). +# - Per-type targets (backend/frontend/shared/…) select the gradual goal for the +# touched code's type; below-target-but-not-below-baseline warns, never blocks (AC5). +# - No baseline committed yet (or a missing/corrupt one) => the gate PRINTS an +# advisory suggestion to stderr and PASSES (bootstrap-only mode), rather than +# blocking everything at 0 (AC4). It does NOT persist the baseline itself. +# - No coverage report measured => fail-safe: BLOCK at red, WARN at lower tiers, +# never a silent pass (edge case). +# +# PERSISTENCE (read before relying on the guardrail): the guardrail is LIVE only +# once a human COMMITS a `baseline.=NN` line to the coverage config. The gate +# never persists the baseline itself — a CI checkout is ephemeral, so any file it +# wrote would be discarded when the runner is torn down (the failure mode: coverage +# could drift down run after run, each run "bootstrapping" a new, lower baseline it +# then throws away, and the regression guard would never fire). So bootstrapping is +# ADVISORY: the gate echoes the suggested `baseline.=NN` to stderr for a human +# to copy into the committed config, and passes without blocking until that commit +# lands. Automated commit-back of a bootstrapped baseline is provider-specific and +# tracked separately — see story #372. +# +# See: +# .pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md +# .pair/knowledge/assets/coverage-config-example.md +# +# Usage (sourced — as in the pipeline / the smoke test): +# source coverage-gate.sh +# coverage_gate "$TIER" "$TYPE" "$COVERAGE_PCT" .pair/adoption/tech/coverage-baseline.md || exit 1 +# Usage (executed directly as a CLI): +# bash coverage-gate.sh "$TIER" "$TYPE" "$COVERAGE_PCT" || exit 1 + +# COVERAGE_DEFAULT_TARGET — advisory fallback when neither target. nor +# target.default is configured (0 => no absolute goal; the baseline guardrail still applies). +: "${COVERAGE_DEFAULT_TARGET:=0}" + +# _cov_is_num — true iff val is a non-negative integer or decimal. +_cov_is_num() { printf '%s' "${1:-}" | grep -Eq '^[0-9]+([.][0-9]+)?$'; } + +# _cov_key_re — escape regex metachars so a dotted key matches literally. +_cov_key_re() { printf '%s' "$1" | sed 's/[.[\*^$]/\\&/g'; } + +# cov_config_value [default] — echo the value of the first `key=value` +# line in the config, ignoring surrounding markdown (fences, headings). Trailing +# CR (CRLF-authored config) and surrounding whitespace are stripped so a +# `baseline.=NN\r` from a Windows/autocrlf checkout is read as the number NN +# rather than rejected as corrupt. Empty/absent => the default (or empty string). +# Reads config ONLY — no code/diff inspection. +cov_config_value() { + local file="$1" key="$2" def="${3:-}" val + val="$(grep -E "^$(_cov_key_re "$key")=" "$file" 2>/dev/null | head -1 | sed 's/^[^=]*=//' | tr -d '\r' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" + if [ -n "$val" ]; then printf '%s' "$val"; else printf '%s' "$def"; fi +} + +# target_for_type — echo the gradual target for a code type: +# target., else target.default, else $COVERAGE_DEFAULT_TARGET. +target_for_type() { + local cfg="$1" type="$2" v + v="$(cov_config_value "$cfg" "target.$type")" + [ -n "$v" ] && { printf '%s' "$v"; return 0; } + v="$(cov_config_value "$cfg" "target.default")" + [ -n "$v" ] && { printf '%s' "$v"; return 0; } + printf '%s' "$COVERAGE_DEFAULT_TARGET" +} + +# baseline_for_type — echo the committed baseline for a type +# (may be empty if a human has not committed one yet). +baseline_for_type() { cov_config_value "$1" "baseline.$2"; } + +# suggest_baseline — print an ADVISORY baseline suggestion to +# stderr. The gate deliberately does NOT persist it (a CI checkout is ephemeral; +# see the PERSISTENCE note above and story #372). A bootstrapped baseline only +# takes effect once a human COMMITS the printed line to the coverage config. +suggest_baseline() { + echo "coverage-gate: no committed baseline for '$1' — bootstrap-only mode: PASSING without blocking (not persisting; a CI checkout is ephemeral, see #372)." >&2 + echo "coverage-gate: to ACTIVATE the guardrail, commit this line to your coverage config:" >&2 + echo " baseline.$1=$2" >&2 +} + +# coverage_gate — the guardrail. +# Returns 0 (pass) / 1 (block). All human-readable output goes to stderr. +coverage_gate() { + local tier="${1:-}" type="${2:-}" current="${3:-}" cfg="${4:-}" baseline target + + # 1. No coverage measured => fail-safe by tier (never a silent pass). + if ! _cov_is_num "$current"; then + case "$tier" in + red | "") + echo "coverage-gate: coverage NOT MEASURED for '$type' at red tier — failing the gate (no silent pass)" >&2 + return 1 + ;; + *) + echo "coverage-gate: coverage not measured for '$type' — warning only at '$tier' tier (heavier suites are not scheduled below red); never a silent pass" >&2 + return 0 + ;; + esac + fi + + # 2. No valid committed baseline yet => advisory suggestion + pass (bootstrap-only + # mode). The gate does NOT persist it; a human must commit baseline.. + baseline="$(baseline_for_type "$cfg" "$type")" + if ! _cov_is_num "$baseline"; then + suggest_baseline "$type" "$current" + return 0 + fi + + # 3. Regression guardrail: below the baseline blocks, at EVERY tier. + if awk -v a="$current" -v b="$baseline" 'BEGIN { exit !(a + 0 < b + 0) }'; then + echo "coverage-gate: REGRESSION — '$type' coverage ${current}% is below baseline ${baseline}%; blocking merge" >&2 + return 1 + fi + + # 4. At/above baseline => pass. Advisory-warn if still below the gradual target. + target="$(target_for_type "$cfg" "$type")" + if _cov_is_num "$target" && awk -v a="$current" -v t="$target" 'BEGIN { exit !(a + 0 < t + 0) }'; then + echo "coverage-gate: '$type' coverage ${current}% holds the baseline ${baseline}% but is below the gradual target ${target}% (advisory, not blocking)" >&2 + fi + return 0 +} + +# CLI entrypoint when executed directly (not sourced): coverage_gate with the args. +if [ "${BASH_SOURCE[0]:-$0}" = "${0}" ]; then + coverage_gate "$@" +fi diff --git a/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md b/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md index b10285d4..fa78fe76 100644 --- a/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md +++ b/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md @@ -139,6 +139,28 @@ jobs: require_suite e2e "$(grep -q '"test:e2e"' package.json && echo 1 || echo 0)" || exit 1 pnpm test:e2e + # coverage: OPT-IN — this job is emitted ONLY when the project sets + # `Coverage guardrail: enabled` in way-of-working.md (default: absent — no job). + # It is a further opt-in on top of tiering, exactly like tiering is an opt-in on + # top of the full suite: absent the flag, `/setup-gates` does not generate it. + coverage: # regression guardrail — runs from 🟡 (where the unit suite produces a report) + needs: [resolve-tier, unit] + if: ${{ needs.resolve-tier.outputs.tier == 'yellow' || needs.resolve-tier.outputs.tier == 'red' }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: pnpm install --frozen-lockfile + - run: pnpm test:coverage # adopted tool emits a coverage report, e.g. coverage/coverage-summary.json + - run: | + source .pair/knowledge/assets/coverage-gate.sh # reads adoption config + a number, no criteria + CFG=.pair/adoption/tech/coverage-baseline.md # committed by the adopter when the guardrail is enabled + # Extract the measured % from whatever the adopted tool emitted. istanbul example: + COV="$(jq '.total.lines.pct' coverage/coverage-summary.json 2>/dev/null)" + # TYPE is the touched code's type (backend|frontend|shared|…): a constant for a + # single-type repo, or run one coverage_gate call per package report in a monorepo. + # The tier is passed only to choose the fail-safe when no report was measured. + coverage_gate "${{ needs.resolve-tier.outputs.tier }}" "${TYPE:-default}" "$COV" "$CFG" || exit 1 + # Deterministic secret scanning — REQUIRED, unconditional at EVERY tier. # No `if:` — a secret is a secret regardless of the change's risk tier. # See ../../quality-assurance/security/secret-scanning.md (R6.5, D24). @@ -158,6 +180,20 @@ jobs: The `secret-scan` job is required and unconditional — it carries no tier `if:`. It is the deterministic security layer provisioned by `/pair-capability-setup-gates`; the full job template, fail-closed requirement, and allowlist mechanism live in [secret-scanning.md](../../quality-assurance/security/secret-scanning.md) (R6.5, D24). This story does not re-implement it — it only guarantees the tier-aware pipeline keeps it unconditional. +## Coverage guardrail (opt-in regression gate consumed by this pipeline) + +The coverage guardrail is **opt-in, off by default** — the twin of tiering itself. `/pair-capability-setup-gates` generates the `coverage` job **only** when the project sets `Coverage guardrail: enabled` in [way-of-working.md](../../../../adoption/tech/way-of-working.md); absent the flag (the default), no coverage job is emitted, in either the full-suite or the tier-aware pipeline. When enabled, coverage is guarded by a **regression gate that runs as a job inside this same pipeline** — not a second, parallel CI mechanism (AC3). It consumes the coverage report the test run already produced and the `resolve-tier` output, exactly like the other jobs; it introduces no independent orchestration and, like `tier-resolve.sh`, reads config + a number only — it carries **no classification criteria** (D18). + +The `coverage` job (above) sources the shipped, provider-agnostic [`coverage-gate.sh`](../../../assets/coverage-gate.sh) helper and calls `coverage_gate `. Its policy: + +- **Blocks a regression, not an absolute wall.** A PR whose coverage drops **below the committed baseline** fails the gate, at every tier (R7.3, epic AC3). A PR that **maintains or improves** coverage passes — the guardrail never demands a fixed X% be hit on every PR. Below the gradual *target* but still at/above the baseline only **warns**. +- **Baseline + per-type targets live in adoption**, in `tech/coverage-baseline.md` (created when the guardrail is enabled; configurable, with KB-sensible defaults) — see [coverage-config-example.md](../../../assets/coverage-config-example.md). The gate reads whatever coverage number the adopted test tooling produced; **no specific coverage tool is mandated** (istanbul `coverage-summary.json`, LCOV, Cobertura, … — the pipeline extracts the % and passes it in). Per-type targets (`backend`/`frontend`/`shared`/…) let the gate apply the threshold matching the touched code's type (AC5). +- **Baseline is human-committed; bootstrapping is advisory** (AC4). The guardrail is live only once a human commits a `baseline.=NN` line to the config. With no committed baseline for a type — or a missing/corrupt one — the gate runs in **bootstrap-only mode**: it prints the current coverage as a suggested `baseline.=NN` to **stderr** and **passes** without blocking. It does **not** persist the baseline itself: a CI checkout is ephemeral, so a written baseline would be discarded and coverage could drift down run after run with the guard never firing. A human copies the suggested line into the committed config to make the guardrail live. Automated commit-back of a bootstrapped baseline is provider-specific and tracked separately — see story #372. +- **Fail-safe on no report**: if no coverage was measured (tooling emitted nothing, or the suite did not run), the gate **blocks at 🔴 red** and **warns at lower tiers** — never a silent pass, matching this pipeline's fail-safe stance. +- **Genuinely untestable surface** (generated files, config) is recorded via the `exclude` key in the config, but that key is **applied by the adopter to their own coverage tool's config** — the gate does **not** read it and this pipeline snippet does **not** pass it. It is documentation of the exclusion intent, not a gate-enforced override. + +When enabled, the `coverage` job is scheduled from 🟡 (where the unit suite that produces the report runs), and 🟢 PRs skip it along with the unit suite; in a full-suite (tiering-off) pipeline with the guardrail enabled, the same job runs on every PR. With the flag absent (the default) there is no coverage job in either pipeline. + ## Required-check wiring (what makes red block merge) Scheduling the jobs is not enough; the code host must **require** them: @@ -195,3 +231,4 @@ jobs: - [github-actions-implementation.md](github-actions-implementation.md) — general GitHub Actions patterns (build, deploy, caching) the templates above draw on. - `/pair-capability-setup-gates` — generates this pipeline for the adopted stack and wires the required checks. - [`tier-resolve.sh`](../../../assets/tier-resolve.sh) — the provider-agnostic, tags-only resolver. +- [`coverage-gate.sh`](../../../assets/coverage-gate.sh) + [coverage-config-example.md](../../../assets/coverage-config-example.md) — the provider-agnostic coverage baseline + regression guardrail and its adoption config format. diff --git a/apps/website/content/docs/concepts/tag-driven-gates.mdx b/apps/website/content/docs/concepts/tag-driven-gates.mdx index c5423252..7068a97a 100644 --- a/apps/website/content/docs/concepts/tag-driven-gates.mdx +++ b/apps/website/content/docs/concepts/tag-driven-gates.mdx @@ -36,6 +36,15 @@ If a PR has **no** `risk:*` tag — or the tag is unknown or malformed — the g If a tier requires a suite the repository does not have (for example, a 🔴 PR with no E2E suite), the gate **fails with an explicit "suite missing" error**. It never silently passes a tier whose checks could not run. +## Coverage guardrail (opt-in, off by default) + +Coverage can be protected by a **baseline regression guardrail** — a job inside the same pipeline, not a separate CI mechanism. It is **opt-in and off by default** (the twin of tiering): a project turns it on by setting `Coverage guardrail: enabled` in its `way-of-working.md`, and `setup-gates` offers it as a guided-setup question (default **No**). Absent the flag, no coverage job is generated. When enabled, a PR that drops coverage **below a committed baseline** fails the gate; a PR that maintains or improves it passes. It is a guardrail against regression, not a wall demanding a fixed percentage on every PR, so a legacy codebase is never failed at 0 on day one. + +- **Baseline and per-type targets live in adoption** (`tech/coverage-baseline.md`, created when the guardrail is enabled), with sensible defaults — different goals for backend, frontend, and shared libraries. No specific coverage tool is required: the gate reads whatever percentage the adopted test tooling emits. +- **The baseline is human-committed.** The guardrail is live only once a `baseline.` line is committed to the config. With none committed (or a missing/corrupt one) the gate runs in **bootstrap-only mode**: it prints a suggested baseline to the CI log and passes, without persisting it — a CI checkout is ephemeral, so it never writes the value itself. A human copies the suggested line into the committed config to activate the guardrail. (Automated commit-back is tracked separately.) +- **The `exclude` config is applied by the adopter** to their own coverage tool — the gate does not read it; it only documents the exclusion intent for genuinely untestable surface. +- **No coverage measured** fails safe: it blocks at 🔴 red and warns at lower tiers — never a silent pass. + ## Post-merge is build + deploy only Once a PR merges to the main branch, the gate does not run again — it already passed. The post-merge staging pipeline performs **build + deploy only**. The gate is the deterministic first filter before merge; review starts only once the gate is green. diff --git a/packages/knowledge-hub/dataset/.pair/adoption/tech/way-of-working.md b/packages/knowledge-hub/dataset/.pair/adoption/tech/way-of-working.md index 6a32aba0..f78b8402 100644 --- a/packages/knowledge-hub/dataset/.pair/adoption/tech/way-of-working.md +++ b/packages/knowledge-hub/dataset/.pair/adoption/tech/way-of-working.md @@ -14,6 +14,7 @@ This document defines the **validated development practices and team workflows** - Quality gate includes: type checking (`ts:check`), testing (`test`), linting (`lint`), formatting (`prettier:fix`). - **Hook manager**: `husky` (KB default, decision D21/Q11) — pre-commit runs fast local checks, pre-push runs lint. Override here if the project uses a different hook manager (e.g. `lefthook`, `simple-git-hooks`); `/pair-capability-setup-gates` reads this override before provisioning. - **Pre-merge tiering**: `disabled` (default) — every PR runs the full pre-merge check suite. Set to `enabled` to opt into risk-tier-scoped pre-merge checks (lighter checks on lower-risk PRs) per [tier-aware-pipeline.md](../../knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md); `/pair-capability-setup-gates` reads this flag before generating the pipeline. +- **Coverage guardrail**: `disabled` (default) — no coverage regression job. Set to `enabled` to opt into a job that blocks a PR whose coverage drops below a human-committed baseline (maintaining/improving passes — not an absolute wall); this also creates `tech/coverage-baseline.md`. See [coverage guardrail](../../knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) + [config format](../../knowledge/assets/coverage-config-example.md); `/pair-capability-setup-gates` reads this flag before generating the pipeline. - **Shared config packages**: see [shared-config-packages.md](../../knowledge/guidelines/code-design/quality-standards/shared-config-packages.md) for the lint/format/type-config pattern and per-type override structure. ### Custom Gate Registry diff --git a/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-config-example.md b/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-config-example.md new file mode 100644 index 00000000..b7d8c390 --- /dev/null +++ b/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-config-example.md @@ -0,0 +1,59 @@ +# `tech/coverage-baseline.md` — Example + +Illustrative coverage config for a project's `tech/coverage-baseline.md` adoption file. This config exists **only when a project opts into the coverage guardrail** (`Coverage guardrail: enabled` in `way-of-working.md`, via `/pair-capability-setup-gates`) — the guardrail is off by default, so a project without it has no such file. Copy the fenced block into your own adoption file and adjust the numbers — this example is committed only as a reference. The [coverage guardrail](../guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) reads this config with the shipped [`coverage-gate.sh`](coverage-gate.sh) helper. + +## What lives here + +Two kinds of value, in one parseable `key=value` block (order-independent; lines are read with a plain `^key=` match, so the surrounding markdown fence and headings are ignored): + +- **`target.`** — the *gradual* per-type coverage goal (human-set). Below the target but not below the baseline only **warns** — it is not a hard wall (R7.3: incremental improvement, not "must hit X% on day one"). +- **`baseline.`** — the *committed* coverage the guardrail protects (**human-committed** — see [Persistence](#persistence-the-baseline-is-human-committed) below). A drop below the baseline **blocks the merge**, at every tier. Leave it unset and the gate runs in **bootstrap-only mode**: it prints the current coverage as a suggested `baseline.=NN` to stderr and passes without blocking, until a human commits that line. +- **`target.default`** — the fallback target for any type without its own `target.` line. +- **`exclude`** — glob(s) for genuinely untestable surface (generated files, config). This is **applied by the adopter to their own coverage tool's config** — the gate does **not** read it and the pipeline snippet does **not** pass it. It is recorded here as the single place the exclusion intent is documented; wiring it into the coverage tool (e.g. istanbul `exclude`, vitest `coverage.exclude`) is the adopter's step. + +The `` is the touched code's type — `backend`, `frontend`, `shared`, or whatever your stack distinguishes; the pipeline passes the type matching the code under test so per-type targets apply (AC5). + +## Persistence: the baseline is human-committed + +The guardrail is **live only once a human commits a `baseline.=NN` line** to this file. The gate never persists a baseline itself: a CI checkout is ephemeral, so anything it wrote would be discarded when the runner is torn down — and coverage could then drift down run after run, each run re-suggesting a new lower baseline it never keeps, with the regression guard never firing. + +So bootstrapping is **advisory**: on an unset/missing/corrupt baseline the gate prints the suggested `baseline.=NN` to stderr and **passes** (bootstrap-only mode). Read the CI log, copy the suggested line into this committed config, and the guardrail becomes live on the next run. Automated commit-back of a bootstrapped baseline is provider-specific and tracked separately — see story **#372**. + +## KB-sensible defaults + +The guardrail is baseline-relative, so absolute targets are advisory. These are sensible starting goals; tune per project: + +| Type | Target | Rationale | +| ---------- | ------ | ------------------------------------------------------ | +| `backend` | 80 | Business logic / services — the highest test priority. | +| `frontend` | 60 | UI — some surface is exercised by E2E, not unit. | +| `shared` | 90 | Reused libraries — a regression here blasts widely. | +| `default` | 70 | Anything without a more specific type. | + +## Example + +```ini +# Coverage config — per-type gradual targets (human-set) + committed baseline. +# Leave a baseline. unset to run in bootstrap-only mode (the gate suggests +# a value on stderr and passes; commit it to activate the guardrail). +target.default=70 +target.backend=80 +target.frontend=60 +target.shared=90 + +# Glob(s) excluded from measurement — applied by the adopter to their coverage +# tool's own config (the gate does not read this key; it is documented here only). +exclude=**/*.generated.ts,**/*.config.ts + +# Committed baseline — the guardrail blocks a drop below these. Human-committed: +# unset/empty => bootstrap-only mode (the gate suggests a value on stderr, passes). +baseline.backend=82.5 +baseline.frontend=61 +# baseline.shared= (unset => bootstrap-only until a human commits a value) +``` + +## Notes + +- **No coverage tool is mandated.** The gate consumes a coverage percentage the pipeline extracts from whatever the adopted test framework emits (istanbul `coverage-summary.json`, LCOV, Cobertura, …). See the guideline's [coverage guardrail](../guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) section for the extraction snippet per stack. +- **Missing/corrupt baseline** runs in bootstrap-only mode: it suggests a value on stderr and passes with a warning — it never blocks everything, and it never persists a value on its own. +- **No coverage measured** (test tooling emitted nothing / tests did not run) fails safe: it blocks at the red tier and warns at lower tiers, never a silent pass. diff --git a/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-gate.sh b/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-gate.sh new file mode 100644 index 00000000..e2bb5bb6 --- /dev/null +++ b/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-gate.sh @@ -0,0 +1,138 @@ +#!/usr/bin/env bash +# coverage-gate.sh — provider-agnostic coverage baseline + regression guardrail +# for the tag-driven pre-merge gate (story #282). +# +# THIS FILE IS A GUARDRAIL, NOT A CLASSIFIER. It reads two inputs only: +# 1. the project's coverage config in adoption (per-type targets + a +# human-committed baseline) — see coverage-config-example.md; and +# 2. a coverage percentage the pipeline already extracted from whatever report +# the adopted test tooling produced (e.g. istanbul coverage-summary.json). +# It contains NO classification criteria (D18): it never inspects the diff, the +# code, file paths, or change size. The risk tier is decided upstream by +# `classify` and passed in only to choose the fail-safe behavior when NO coverage +# was measured (block at red, warn at lower tiers). +# +# Policy (story #282): +# - The guardrail blocks a REGRESSION below the committed baseline, at every +# tier — not "must hit X% absolute". Maintaining or improving passes (AC1/AC2). +# - Per-type targets (backend/frontend/shared/…) select the gradual goal for the +# touched code's type; below-target-but-not-below-baseline warns, never blocks (AC5). +# - No baseline committed yet (or a missing/corrupt one) => the gate PRINTS an +# advisory suggestion to stderr and PASSES (bootstrap-only mode), rather than +# blocking everything at 0 (AC4). It does NOT persist the baseline itself. +# - No coverage report measured => fail-safe: BLOCK at red, WARN at lower tiers, +# never a silent pass (edge case). +# +# PERSISTENCE (read before relying on the guardrail): the guardrail is LIVE only +# once a human COMMITS a `baseline.=NN` line to the coverage config. The gate +# never persists the baseline itself — a CI checkout is ephemeral, so any file it +# wrote would be discarded when the runner is torn down (the failure mode: coverage +# could drift down run after run, each run "bootstrapping" a new, lower baseline it +# then throws away, and the regression guard would never fire). So bootstrapping is +# ADVISORY: the gate echoes the suggested `baseline.=NN` to stderr for a human +# to copy into the committed config, and passes without blocking until that commit +# lands. Automated commit-back of a bootstrapped baseline is provider-specific and +# tracked separately — see story #372. +# +# See: +# .pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md +# .pair/knowledge/assets/coverage-config-example.md +# +# Usage (sourced — as in the pipeline / the smoke test): +# source coverage-gate.sh +# coverage_gate "$TIER" "$TYPE" "$COVERAGE_PCT" .pair/adoption/tech/coverage-baseline.md || exit 1 +# Usage (executed directly as a CLI): +# bash coverage-gate.sh "$TIER" "$TYPE" "$COVERAGE_PCT" || exit 1 + +# COVERAGE_DEFAULT_TARGET — advisory fallback when neither target. nor +# target.default is configured (0 => no absolute goal; the baseline guardrail still applies). +: "${COVERAGE_DEFAULT_TARGET:=0}" + +# _cov_is_num — true iff val is a non-negative integer or decimal. +_cov_is_num() { printf '%s' "${1:-}" | grep -Eq '^[0-9]+([.][0-9]+)?$'; } + +# _cov_key_re — escape regex metachars so a dotted key matches literally. +_cov_key_re() { printf '%s' "$1" | sed 's/[.[\*^$]/\\&/g'; } + +# cov_config_value [default] — echo the value of the first `key=value` +# line in the config, ignoring surrounding markdown (fences, headings). Trailing +# CR (CRLF-authored config) and surrounding whitespace are stripped so a +# `baseline.=NN\r` from a Windows/autocrlf checkout is read as the number NN +# rather than rejected as corrupt. Empty/absent => the default (or empty string). +# Reads config ONLY — no code/diff inspection. +cov_config_value() { + local file="$1" key="$2" def="${3:-}" val + val="$(grep -E "^$(_cov_key_re "$key")=" "$file" 2>/dev/null | head -1 | sed 's/^[^=]*=//' | tr -d '\r' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//')" + if [ -n "$val" ]; then printf '%s' "$val"; else printf '%s' "$def"; fi +} + +# target_for_type — echo the gradual target for a code type: +# target., else target.default, else $COVERAGE_DEFAULT_TARGET. +target_for_type() { + local cfg="$1" type="$2" v + v="$(cov_config_value "$cfg" "target.$type")" + [ -n "$v" ] && { printf '%s' "$v"; return 0; } + v="$(cov_config_value "$cfg" "target.default")" + [ -n "$v" ] && { printf '%s' "$v"; return 0; } + printf '%s' "$COVERAGE_DEFAULT_TARGET" +} + +# baseline_for_type — echo the committed baseline for a type +# (may be empty if a human has not committed one yet). +baseline_for_type() { cov_config_value "$1" "baseline.$2"; } + +# suggest_baseline — print an ADVISORY baseline suggestion to +# stderr. The gate deliberately does NOT persist it (a CI checkout is ephemeral; +# see the PERSISTENCE note above and story #372). A bootstrapped baseline only +# takes effect once a human COMMITS the printed line to the coverage config. +suggest_baseline() { + echo "coverage-gate: no committed baseline for '$1' — bootstrap-only mode: PASSING without blocking (not persisting; a CI checkout is ephemeral, see #372)." >&2 + echo "coverage-gate: to ACTIVATE the guardrail, commit this line to your coverage config:" >&2 + echo " baseline.$1=$2" >&2 +} + +# coverage_gate — the guardrail. +# Returns 0 (pass) / 1 (block). All human-readable output goes to stderr. +coverage_gate() { + local tier="${1:-}" type="${2:-}" current="${3:-}" cfg="${4:-}" baseline target + + # 1. No coverage measured => fail-safe by tier (never a silent pass). + if ! _cov_is_num "$current"; then + case "$tier" in + red | "") + echo "coverage-gate: coverage NOT MEASURED for '$type' at red tier — failing the gate (no silent pass)" >&2 + return 1 + ;; + *) + echo "coverage-gate: coverage not measured for '$type' — warning only at '$tier' tier (heavier suites are not scheduled below red); never a silent pass" >&2 + return 0 + ;; + esac + fi + + # 2. No valid committed baseline yet => advisory suggestion + pass (bootstrap-only + # mode). The gate does NOT persist it; a human must commit baseline.. + baseline="$(baseline_for_type "$cfg" "$type")" + if ! _cov_is_num "$baseline"; then + suggest_baseline "$type" "$current" + return 0 + fi + + # 3. Regression guardrail: below the baseline blocks, at EVERY tier. + if awk -v a="$current" -v b="$baseline" 'BEGIN { exit !(a + 0 < b + 0) }'; then + echo "coverage-gate: REGRESSION — '$type' coverage ${current}% is below baseline ${baseline}%; blocking merge" >&2 + return 1 + fi + + # 4. At/above baseline => pass. Advisory-warn if still below the gradual target. + target="$(target_for_type "$cfg" "$type")" + if _cov_is_num "$target" && awk -v a="$current" -v t="$target" 'BEGIN { exit !(a + 0 < t + 0) }'; then + echo "coverage-gate: '$type' coverage ${current}% holds the baseline ${baseline}% but is below the gradual target ${target}% (advisory, not blocking)" >&2 + fi + return 0 +} + +# CLI entrypoint when executed directly (not sourced): coverage_gate with the args. +if [ "${BASH_SOURCE[0]:-$0}" = "${0}" ]; then + coverage_gate "$@" +fi diff --git a/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md b/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md index 2b952dbb..d709a546 100644 --- a/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md +++ b/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md @@ -139,6 +139,28 @@ jobs: require_suite e2e "$(grep -q '"test:e2e"' package.json && echo 1 || echo 0)" || exit 1 pnpm test:e2e + # coverage: OPT-IN — this job is emitted ONLY when the project sets + # `Coverage guardrail: enabled` in way-of-working.md (default: absent — no job). + # It is a further opt-in on top of tiering, exactly like tiering is an opt-in on + # top of the full suite: absent the flag, `/setup-gates` does not generate it. + coverage: # regression guardrail — runs from 🟡 (where the unit suite produces a report) + needs: [resolve-tier, unit] + if: ${{ needs.resolve-tier.outputs.tier == 'yellow' || needs.resolve-tier.outputs.tier == 'red' }} + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: pnpm install --frozen-lockfile + - run: pnpm test:coverage # adopted tool emits a coverage report, e.g. coverage/coverage-summary.json + - run: | + source .pair/knowledge/assets/coverage-gate.sh # reads adoption config + a number, no criteria + CFG=.pair/adoption/tech/coverage-baseline.md # committed by the adopter when the guardrail is enabled + # Extract the measured % from whatever the adopted tool emitted. istanbul example: + COV="$(jq '.total.lines.pct' coverage/coverage-summary.json 2>/dev/null)" + # TYPE is the touched code's type (backend|frontend|shared|…): a constant for a + # single-type repo, or run one coverage_gate call per package report in a monorepo. + # The tier is passed only to choose the fail-safe when no report was measured. + coverage_gate "${{ needs.resolve-tier.outputs.tier }}" "${TYPE:-default}" "$COV" "$CFG" || exit 1 + # Deterministic secret scanning — REQUIRED, unconditional at EVERY tier. # No `if:` — a secret is a secret regardless of the change's risk tier. # See ../../quality-assurance/security/secret-scanning.md (R6.5, D24). @@ -158,6 +180,20 @@ jobs: The `secret-scan` job is required and unconditional — it carries no tier `if:`. It is the deterministic security layer provisioned by `/setup-gates`; the full job template, fail-closed requirement, and allowlist mechanism live in [secret-scanning.md](../../quality-assurance/security/secret-scanning.md) (R6.5, D24). This story does not re-implement it — it only guarantees the tier-aware pipeline keeps it unconditional. +## Coverage guardrail (opt-in regression gate consumed by this pipeline) + +The coverage guardrail is **opt-in, off by default** — the twin of tiering itself. `/setup-gates` generates the `coverage` job **only** when the project sets `Coverage guardrail: enabled` in [way-of-working.md](../../../../adoption/tech/way-of-working.md); absent the flag (the default), no coverage job is emitted, in either the full-suite or the tier-aware pipeline. When enabled, coverage is guarded by a **regression gate that runs as a job inside this same pipeline** — not a second, parallel CI mechanism (AC3). It consumes the coverage report the test run already produced and the `resolve-tier` output, exactly like the other jobs; it introduces no independent orchestration and, like `tier-resolve.sh`, reads config + a number only — it carries **no classification criteria** (D18). + +The `coverage` job (above) sources the shipped, provider-agnostic [`coverage-gate.sh`](../../../assets/coverage-gate.sh) helper and calls `coverage_gate `. Its policy: + +- **Blocks a regression, not an absolute wall.** A PR whose coverage drops **below the committed baseline** fails the gate, at every tier (R7.3, epic AC3). A PR that **maintains or improves** coverage passes — the guardrail never demands a fixed X% be hit on every PR. Below the gradual *target* but still at/above the baseline only **warns**. +- **Baseline + per-type targets live in adoption**, in `tech/coverage-baseline.md` (created when the guardrail is enabled; configurable, with KB-sensible defaults) — see [coverage-config-example.md](../../../assets/coverage-config-example.md). The gate reads whatever coverage number the adopted test tooling produced; **no specific coverage tool is mandated** (istanbul `coverage-summary.json`, LCOV, Cobertura, … — the pipeline extracts the % and passes it in). Per-type targets (`backend`/`frontend`/`shared`/…) let the gate apply the threshold matching the touched code's type (AC5). +- **Baseline is human-committed; bootstrapping is advisory** (AC4). The guardrail is live only once a human commits a `baseline.=NN` line to the config. With no committed baseline for a type — or a missing/corrupt one — the gate runs in **bootstrap-only mode**: it prints the current coverage as a suggested `baseline.=NN` to **stderr** and **passes** without blocking. It does **not** persist the baseline itself: a CI checkout is ephemeral, so a written baseline would be discarded and coverage could drift down run after run with the guard never firing. A human copies the suggested line into the committed config to make the guardrail live. Automated commit-back of a bootstrapped baseline is provider-specific and tracked separately — see story #372. +- **Fail-safe on no report**: if no coverage was measured (tooling emitted nothing, or the suite did not run), the gate **blocks at 🔴 red** and **warns at lower tiers** — never a silent pass, matching this pipeline's fail-safe stance. +- **Genuinely untestable surface** (generated files, config) is recorded via the `exclude` key in the config, but that key is **applied by the adopter to their own coverage tool's config** — the gate does **not** read it and this pipeline snippet does **not** pass it. It is documentation of the exclusion intent, not a gate-enforced override. + +When enabled, the `coverage` job is scheduled from 🟡 (where the unit suite that produces the report runs), and 🟢 PRs skip it along with the unit suite; in a full-suite (tiering-off) pipeline with the guardrail enabled, the same job runs on every PR. With the flag absent (the default) there is no coverage job in either pipeline. + ## Required-check wiring (what makes red block merge) Scheduling the jobs is not enough; the code host must **require** them: @@ -195,3 +231,4 @@ jobs: - [github-actions-implementation.md](github-actions-implementation.md) — general GitHub Actions patterns (build, deploy, caching) the templates above draw on. - `/setup-gates` — generates this pipeline for the adopted stack and wires the required checks. - [`tier-resolve.sh`](../../../assets/tier-resolve.sh) — the provider-agnostic, tags-only resolver. +- [`coverage-gate.sh`](../../../assets/coverage-gate.sh) + [coverage-config-example.md](../../../assets/coverage-config-example.md) — the provider-agnostic coverage baseline + regression guardrail and its adoption config format. diff --git a/packages/knowledge-hub/dataset/.skills/capability/setup-gates/SKILL.md b/packages/knowledge-hub/dataset/.skills/capability/setup-gates/SKILL.md index 3decb229..2deaf19e 100644 --- a/packages/knowledge-hub/dataset/.skills/capability/setup-gates/SKILL.md +++ b/packages/knowledge-hub/dataset/.skills/capability/setup-gates/SKILL.md @@ -50,7 +50,7 @@ Configure CI/CD quality gates for the project. Reads quality assurance guideline 1. **Act**: Read quality assurance guidelines: - [quality-assurance.md](../../../.pair/knowledge/guidelines/technical-standards/git-workflow/quality-assurance.md) — gate types and checklists - [quality-gates.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-standards/quality-gates.md) — gate framework and registry format - - [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md) — the **opt-in** optimization that reduces checks per risk tier (tags only, fail-safe red); default is the full suite on every PR, tier-aware only when the project opts in + - [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md) — the **opt-in** optimization that reduces checks per risk tier (tags only, fail-safe red); default is the full suite on every PR, tier-aware only when the project opts in. Also defines the **opt-in coverage guardrail** job (off by default) and its [config format](../../../.pair/knowledge/assets/coverage-config-example.md) - [quality-model.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-model.md) §4 — the single source of the tier→checks matrix the pipeline projects (never re-declared in generated config) - [shared-config-packages.md](../../../.pair/knowledge/guidelines/code-design/quality-standards/shared-config-packages.md) — shared-config-package pattern, per-type overrides, `tools/*` reference implementation - [secret-scanning.md](../../../.pair/knowledge/guidelines/quality-assurance/security/secret-scanning.md) — the deterministic, required-at-every-tier CI job this skill provisions (R6.5, D24) — never a skill, never an LLM judgment call @@ -98,7 +98,16 @@ Configure CI/CD quality gates for the project. Reads quality assurance guideline - **Quick mode / non-interactive / CI** — take the default: **No** (full pipeline). Never prompt. - Record the resolved choice for Step 4: `TIERING = enabled | disabled` (default `disabled`). -4. **Verify**: Developer approves gate configuration and the tiering choice is resolved (`disabled` unless explicitly opted in). +4. **Act — coverage guardrail opt-in** (same [guided/quick convention](../../../.pair/knowledge/guidelines/technical-standards/ai-development/skill-conventions/guided-quick-setup.md), default **No** — the twin of the tiering opt-in above): the coverage regression guardrail is off by default; a project opts in like any other input. + - **Guided mode** (interactive/TTY) — ask **one** question, pre-filled with the default: + + > Enable the coverage guardrail? This adds a job that blocks a PR whose test coverage drops **below a committed baseline** (maintaining/improving passes — it is not an absolute wall). Off by default; the baseline is human-committed and bootstraps advisory-only on first run. **[default: No]** + + Default (Enter) ⇒ **No** (no coverage job). Only an explicit "Yes" opts in. + - **Quick mode / non-interactive / CI** — take the default: **No**. Never prompt. + - Record the resolved choice for Step 4: `COVERAGE_GUARDRAIL = enabled | disabled` (default `disabled`). + +5. **Verify**: Developer approves gate configuration; the tiering choice (`disabled` unless opted in) and the coverage-guardrail choice (`disabled` unless opted in) are both resolved. ### Step 4: Write Configuration @@ -106,13 +115,15 @@ Configure CI/CD quality gates for the project. Reads quality assurance guideline - Set quality gate command (e.g., `pnpm quality-gate`) - Write or update Custom Gate Registry table - **Pre-merge tiering flag**: if `TIERING = enabled` (Step 3.3 opt-in), record `Pre-merge tiering: enabled` in the Quality Gates section. If `disabled` (the default), either omit the line or record `Pre-merge tiering: disabled` — both mean full-suite. Never write `enabled` without the explicit opt-in. + - **Coverage guardrail flag**: if `COVERAGE_GUARDRAIL = enabled` (Step 3.4 opt-in), record `Coverage guardrail: enabled` in the Quality Gates section; if `disabled` (the default), either omit the line or record `Coverage guardrail: disabled`. Never write `enabled` without the explicit opt-in. **When enabled**, also create `tech/coverage-baseline.md` from the KB template ([coverage-config-example.md](../../../.pair/knowledge/assets/coverage-config-example.md)) with the per-type targets for the adopted stack and the baselines left unset (bootstrap-only until a human commits them — see that example's Persistence note). **Idempotent**: if the flag is already `enabled` and `tech/coverage-baseline.md` already exists, confirm rather than overwrite. 2. **Act**: Generate the **pre-merge pipeline** appropriate for the adopted stack and hosting. **Which pipeline depends on `TIERING`:** - GitHub Actions → `.github/workflows/quality.yml` - GitLab → `.gitlab-ci.yml` quality stage - Other → document commands for manual pipeline setup - **`TIERING = disabled` (default) → full-suite pipeline**: every PR runs base (install + lint + type + build) **and** unit **and** integration/E2E **and** coverage — all jobs unconditional, no tag resolution, no `risk:*` reads. This is the safe default; do **not** generate the tier-aware jobs. - **`TIERING = enabled` (opt-in) → tier-aware pipeline** following [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md): the pipeline **reads the PR's `risk:*` tag only** (via [`tier-resolve.sh`](../../../.pair/knowledge/assets/tier-resolve.sh) — tags only, no classification criteria of its own, D18) and schedules the [quality-model.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-model.md) §4 matrix: base every tier, unit from 🟡, integration/E2E from 🔴; **untagged/unknown/malformed ⇒ 🔴** (fail-safe); a required suite missing at a tier ⇒ **explicit failure** (`require_suite`), never a silent pass. Do not copy any tier threshold into the generated config — it is a projection of the model, not a second source. - - **Wire required checks**: mark `base` and `secret-scan` as required status checks on the protected branch. In full-suite mode all suite jobs are required; in tier-aware mode the tier-scoped jobs are required-when-scheduled (or via one aggregating gate job). A red gate blocks merge either way. + - **Coverage job (opt-in)**: emit the `coverage` job from [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md#coverage-guardrail-opt-in-regression-gate-consumed-by-this-pipeline) **only when `COVERAGE_GUARDRAIL = enabled`** (Step 3.4). It sources [`coverage-gate.sh`](../../../.pair/knowledge/assets/coverage-gate.sh) (config + a number only, no criteria — D18), reads `tech/coverage-baseline.md`, and blocks a regression below the committed baseline (maintain/improve passes; bootstrap-only when unset). With the flag absent (default) do **not** emit a coverage job in either pipeline mode. + - **Wire required checks**: mark `base` and `secret-scan` as required status checks on the protected branch. In full-suite mode all suite jobs are required; in tier-aware mode the tier-scoped jobs are required-when-scheduled (or via one aggregating gate job). A red gate blocks merge either way. The coverage job (when enabled) is required-when-scheduled like the other tier-scoped jobs. - **Post-merge staging** (on merge to the main branch): **build + deploy only, no gate re-run** — the gate already passed pre-merge (identical in both modes). 3. **Act**: Write the secret-scanning job — required, not an optional proposal line, in **both** pipeline modes: resolve the scanner (Argument > Adoption's Custom Gate Registry override > KB default gitleaks), then write the [CI Job Template](../../../.pair/knowledge/guidelines/quality-assurance/security/secret-scanning.md#ci-job-template-github-actions) into the same pipeline file as a `required` job, and provision a starting `.gitleaks.toml` at the project root (see the [allowlist mechanism](../../../.pair/knowledge/guidelines/quality-assurance/security/secret-scanning.md#allowlist-mechanism-adoption-controlled)) if one doesn't already exist. Never write the job with `continue-on-error` — fail-closed is not optional (R6.5). 4. **Verify**: Configuration files written, including the pre-merge pipeline (full-suite by default; tag-only tier-aware resolution + tier-scoped jobs only when `TIERING = enabled`) + required-check wiring, the post-merge staging (build+deploy only) config, the secret-scanning job, and `.gitleaks.toml`. If tiering is enabled, grep the generated pipeline for classification tokens (schema/diff/path heuristics) — there must be none (D18). If disabled, confirm no `risk:*` resolution was generated and every suite job runs unconditionally. @@ -152,6 +163,7 @@ GATE CONFIGURATION COMPLETE: ├── Pre-production: [N gates configured | N/A] ├── Pipeline: [file path | manual — full-suite pre-merge (default: every PR runs all suites) OR tier-aware pre-merge (opt-in: tags only, fail-safe 🔴) + post-merge staging build+deploy] ├── Pre-merge tiering: [disabled (default — full suite every PR) | enabled (risk-tier-scoped, opt-in)] +├── Coverage guardrail: [disabled (default — no coverage job) | enabled (opt-in — coverage job + tech/coverage-baseline.md created)] ├── Required checks: [base + secret-scan required; full-suite: all suite jobs required | tier-aware: tier-scoped jobs required-when-scheduled] ├── Secret Scan: [gitleaks (KB default) | — job + .gitleaks.toml written | already configured] ├── Shared configs: [package list | N/A — non-JS, documented pointer only] @@ -184,6 +196,7 @@ See [graceful degradation](../../../.pair/knowledge/guidelines/technical-standar - This skill **modifies files** — it writes to way-of-working.md, creates/updates CI/CD pipeline configuration (the pre-merge pipeline — full-suite by default, tier-aware only on opt-in — + post-merge staging + the required secret-scanning job and `.gitleaks.toml`), and provisions shared lint/format config packages + hook manager files (`.husky/` by default). - **Full checks are the default; risk-tier reduction is opt-in** (ADL 2026-07-20): unless the project sets `Pre-merge tiering: enabled` in way-of-working.md (Step 3.3 opt-in), the generated pipeline runs the **full suite on every PR** — the safe, pre-tiering behavior. Tiering only *reduces* checks on lower-risk PRs, so it must be enabled explicitly. +- **Coverage guardrail is opt-in, off by default** (story #282): the `coverage` regression job and `tech/coverage-baseline.md` are generated **only** when the project sets `Coverage guardrail: enabled` (Step 3.4 opt-in) — the twin of the tiering opt-in. Absent the flag, no coverage job and no baseline file. The baseline is human-committed; the gate bootstraps advisory-only (suggests on stderr, never persists — a CI checkout is ephemeral; automated commit-back is story #372). See [coverage-config-example.md](../../../.pair/knowledge/assets/coverage-config-example.md). - **When tiering is enabled, the generated pipeline reads classification tags only, never classifies** (D18): tier criteria live in [quality-model.md](../../../.pair/knowledge/guidelines/quality-assurance/quality-model.md) §3/§4; the pipeline is the deterministic Automation layer that projects the matrix from the PR's `risk:*` tag. Untagged ⇒ 🔴 (fail-safe). See [tier-aware-pipeline.md](../../../.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md). - **Secret scanning is CI config, not a judgment call** (D24, anti-complexity): this skill provisions the job mechanically; it never evaluates whether a diff contains a secret itself — that is gitleaks' (or the adopted scanner's) job at runtime, with no LLM in the loop. `/assess-security` never re-implements this — see that skill's own Notes. - **Idempotent** — see [idempotency convention](../../../.pair/knowledge/guidelines/technical-standards/ai-development/skill-conventions/idempotency.md). This skill's check: an already-configured project (incl. provisioned shared configs and hooks) is confirmed; update only on explicit developer request. Conflicting local config is always resolved by asking first (see Edge Cases above). diff --git a/scripts/smoke-tests/run-all.sh b/scripts/smoke-tests/run-all.sh index 503e8f27..e3ce41a8 100755 --- a/scripts/smoke-tests/run-all.sh +++ b/scripts/smoke-tests/run-all.sh @@ -265,6 +265,7 @@ if [ "$IS_CI" = "true" ]; then "install-preconditions.sh" "default-resolution.sh" "tier-aware-gate.sh" + "coverage-gate.sh" ) for t in "${CI_TESTS[@]}"; do diff --git a/scripts/smoke-tests/scenarios/coverage-gate.sh b/scripts/smoke-tests/scenarios/coverage-gate.sh new file mode 100644 index 00000000..b5632117 --- /dev/null +++ b/scripts/smoke-tests/scenarios/coverage-gate.sh @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# OFFLINE_SAFE=true +# +# Coverage baseline + CI guardrail — verification scenario (story #282). +# +# Exercises the shipped, provider-agnostic coverage guardrail end-to-end and +# audits the guideline for the non-negotiable properties: +# 1. a regression below the established baseline BLOCKS (fails) at every tier; +# 2. maintaining or improving coverage PASSES (guardrail, not an absolute wall); +# 3. per-type targets select the threshold matching the touched code's type; +# 4. baseline bootstrapping is ADVISORY on first run / when missing / when +# corrupt — it SUGGESTS a value on stderr and passes without persisting +# (a CI checkout is ephemeral; the baseline is human-committed, see #372), +# instead of blocking everything at 0; +# 5. a missing coverage report fails safe: BLOCKS at red tier, WARNS at lower +# tiers, never a silent pass; +# 6. the guardrail reads adoption config + a coverage number only — it carries NO +# classification criteria (D18): it never inspects the diff, code, or paths. +# +# Per the gate-tooling ADL (2026-07-13) this shell/asset surface is verified with a +# smoke test, not a vitest unit test. +source "$(dirname "$0")/../lib/utils.sh" +ensure_tmp_dir + +TEST_NAME="Coverage Baseline + CI Guardrail" +echo "=== Running $TEST_NAME ===" + +REPO_ROOT="${REPO_ROOT:-$(cd "$(dirname "$0")/../../.." && pwd)}" +GATE="$REPO_ROOT/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-gate.sh" +GUIDELINE="$REPO_ROOT/packages/knowledge-hub/dataset/.pair/knowledge/guidelines/infrastructure/cicd-strategy/tier-aware-pipeline.md" +EXAMPLE="$REPO_ROOT/packages/knowledge-hub/dataset/.pair/knowledge/assets/coverage-config-example.md" + +FAILED=0 +check() { # check + if [ "$2" = "$3" ]; then log_succ "$1 => $3"; else log_fail "$1: expected '$2' got '$3'"; FAILED=1; fi +} +pass() { # pass — expects exit 0 + local desc="$1"; shift + if "$@" >/dev/null 2>&1; then log_succ "$desc (passed as expected)"; else log_fail "$desc: expected pass (exit 0) got fail"; FAILED=1; fi +} +block() { # block — expects non-zero exit + local desc="$1"; shift + if "$@" >/dev/null 2>&1; then log_fail "$desc: expected BLOCK (exit!=0) got pass"; FAILED=1; else log_succ "$desc (blocked as expected)"; fi +} + +assert_file "$GATE" || exit 1 +assert_file "$GUIDELINE" || exit 1 +assert_file "$EXAMPLE" || exit 1 + +# shellcheck source=/dev/null +source "$GATE" + +# --- Config fixture: per-type targets + an established baseline (parseable lines +# that also live verbatim inside the adoption markdown's fenced config block) --- +CFG="$TMP_DIR/coverage-baseline.md" +cat >"$CFG" <<'EOF' +# Coverage config (fixture) +```ini +target.default=70 +target.backend=80 +target.frontend=60 +target.shared=90 +exclude=**/*.generated.ts +baseline.default=75 +baseline.backend=82.5 +baseline.frontend=61 +``` +EOF + +# --- Per-type target selection (AC5) --- +check "target backend" 80 "$(target_for_type "$CFG" backend)" +check "target frontend" 60 "$(target_for_type "$CFG" frontend)" +check "target shared" 90 "$(target_for_type "$CFG" shared)" +check "target unknown->default" 70 "$(target_for_type "$CFG" mystery)" + +# --- Baseline read --- +check "baseline backend" 82.5 "$(baseline_for_type "$CFG" backend)" + +# --- CRLF-authored config: a trailing CR must NOT corrupt a human-committed +# baseline (Minor fix — Windows/autocrlf checkout). baseline.backend=82.5\r must +# read as 82.5, so 80 < 82.5 still BLOCKS instead of silently re-bootstrapping. --- +CRLF="$TMP_DIR/coverage-crlf.md" +printf 'target.backend=80\r\nbaseline.backend=82.5\r\n' >"$CRLF" +check "CRLF baseline read as number" 82.5 "$(baseline_for_type "$CRLF" backend)" +block "CRLF baseline honored (80 < 82.5 blocks)" coverage_gate red backend 80 "$CRLF" + +# --- Regression blocks; maintain/improve passes (AC1/AC2) — at EVERY tier --- +block "green: backend 80 < baseline 82.5 blocks" coverage_gate green backend 80 "$CFG" +block "yellow: backend 80 < baseline 82.5 blocks" coverage_gate yellow backend 80 "$CFG" +block "red: backend 80 < baseline 82.5 blocks" coverage_gate red backend 80 "$CFG" +pass "backend 82.5 == baseline passes" coverage_gate yellow backend 82.5 "$CFG" +pass "backend 90 > baseline passes" coverage_gate yellow backend 90 "$CFG" +pass "frontend 61 == baseline passes" coverage_gate yellow frontend 61 "$CFG" + +# --- Missing coverage report: fail-safe by tier (edge case) --- +block "red + no report blocks (fail-safe)" coverage_gate red backend "" "$CFG" +pass "yellow + no report warns (no block)" coverage_gate yellow backend "" "$CFG" +pass "green + no report warns (no block)" coverage_gate green backend "" "$CFG" +# missing report must warn on stderr (never silently pass) +if coverage_gate yellow backend "" "$CFG" 2>&1 >/dev/null | grep -qi 'not measured\|no coverage'; then + log_succ "missing report warns on stderr" +else + log_fail "missing report did not warn"; FAILED=1 +fi + +# --- Baseline bootstrapping is ADVISORY (AC4 + persistence=A): no committed +# baseline => the gate SUGGESTS a value on stderr and PASSES, but does NOT persist +# it (a CI checkout is ephemeral; a human commits it — see #372). --- +BOOT="$TMP_DIR/coverage-bootstrap.md" +cat >"$BOOT" <<'EOF' +```ini +target.default=70 +``` +EOF +BOOT_BEFORE="$(cat "$BOOT")" +pass "no baseline -> advisory pass, does not block" coverage_gate red backend 55 "$BOOT" +if coverage_gate red backend 55 "$BOOT" 2>&1 >/dev/null | grep -q 'baseline.backend=55'; then + log_succ "bootstrap SUGGESTS baseline.backend=55 on stderr" +else + log_fail "bootstrap did not suggest a baseline on stderr"; FAILED=1 +fi +if [ "$(cat "$BOOT")" = "$BOOT_BEFORE" ]; then + log_succ "bootstrap did NOT persist to the config (advisory-only; ephemeral-safe)" +else + log_fail "bootstrap wrote to the config (must be advisory-only)"; FAILED=1 +fi + +# --- Corrupt baseline is advisory too: pass without blocking AND without +# overwriting the (corrupt) committed value on its own (edge case). --- +CORRUPT="$TMP_DIR/coverage-corrupt.md" +cat >"$CORRUPT" <<'EOF' +```ini +baseline.backend=not-a-number +``` +EOF +CORRUPT_BEFORE="$(cat "$CORRUPT")" +pass "corrupt baseline -> advisory pass, no block" coverage_gate red backend 70 "$CORRUPT" +if [ "$(cat "$CORRUPT")" = "$CORRUPT_BEFORE" ]; then + log_succ "corrupt baseline NOT overwritten by the gate (advisory-only)" +else + log_fail "corrupt baseline was overwritten (must be advisory-only)"; FAILED=1 +fi + +# --- Grep audit: the gate carries NO classification criteria (D18). It reads the +# adoption config and a coverage number — it never inspects the diff/code/paths. --- +CRITERIA='\b(git diff|--numstat|files?[ -]changed|risk:(green|yellow|red)|schema|migration)\b' +if grep -Eiq "$CRITERIA" "$GATE"; then + log_fail "coverage gate leaks classification criteria"; grep -Ein "$CRITERIA" "$GATE"; FAILED=1 +else + log_succ "coverage gate contains no classification criteria" +fi + +# --- Guideline audit: the coverage guardrail is documented as a job consumed by +# THIS pipeline (AC3), baseline-relative (regression), with per-type targets and +# fail-safe behavior. --- +if grep -qi 'coverage guardrail' "$GUIDELINE" \ + && grep -qi 'baseline' "$GUIDELINE" \ + && grep -qi 'regression' "$GUIDELINE"; then + log_succ "guideline documents the coverage guardrail (baseline/regression)" +else + log_fail "guideline missing coverage guardrail documentation"; FAILED=1 +fi + +# --- Opt-in audit (default off): the guardrail must be documented as opt-in in +# both the guideline and the docs, and its persistence model must be advisory / +# human-committed — the Major fix (persistence=A). --- +if grep -qi 'opt-in' "$GUIDELINE" \ + && grep -Eqi 'human-committed|human commits|bootstrap-only' "$GUIDELINE"; then + log_succ "guideline documents opt-in + advisory/human-committed persistence" +else + log_fail "guideline missing opt-in / persistence documentation"; FAILED=1 +fi + +# --- Example audit: persistence=A (human-committed, advisory, commit-back #372) +# and the exclude-is-adopter-applied clarification (the two remaining Minors). --- +if grep -Eqi 'human-committed|bootstrap-only' "$EXAMPLE" \ + && grep -q '#372' "$EXAMPLE" \ + && grep -qi 'applied by the adopter' "$EXAMPLE"; then + log_succ "example documents persistence=A (#372) + exclude adopter-applied" +else + log_fail "example missing persistence / exclude clarification"; FAILED=1 +fi + +# --- "machine-maintained" wording must be gone from the shipped assets (Major fix). --- +if grep -qi 'machine-maintained' "$GATE" "$EXAMPLE" "$GUIDELINE"; then + log_fail "'machine-maintained' wording still present (should be removed)"; FAILED=1 +else + log_succ "no misleading 'machine-maintained' wording remains" +fi + +if [ "$FAILED" -ne 0 ]; then + log_fail "$TEST_NAME had failures"; exit 1 +fi +log_succ "$TEST_NAME passed"