Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,24 @@ drift and reports stale entries when old violations are fixed. See the
[Baseline guide](https://skillsaw.org/baseline/) for matching behavior,
ratchet behavior, and refresh workflow.

### Lint only your changes

`--since` builds the baseline on the fly from git history — no committed
`.skillsaw-baseline.json` needed. It lints the merge-base of HEAD and the
given ref in a temporary git worktree and reports only the violations
introduced since:

```bash
# Report only violations introduced on your branch
skillsaw lint --since origin/main
```

Pre-existing violations stay suppressed even when your change shifts the
lines around them, and ratchet rules (like `context-budget`) only fire when
your change makes the tracked value worse. See
[Lint only your changes](https://skillsaw.org/baseline/#lint-only-your-changes-since)
for the mechanism and caveats.

## CI Integration

```yaml
Expand Down
8 changes: 8 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@ inputs:
description: 'Skip custom rules defined in .skillsaw.yaml (recommended for CI on untrusted PRs)'
required: false
default: 'true'
since:
description: 'Only report violations introduced since the merge-base of HEAD and this ref (e.g. origin/main). Requires git history: use actions/checkout with fetch-depth: 0 (or fetch the base branch).'
required: false
default: ''

outputs:
exit-code:
Expand Down Expand Up @@ -86,12 +90,16 @@ runs:
SKILLSAW_FAIL_ON: ${{ inputs.fail-on }}
SKILLSAW_VERBOSE: ${{ inputs.verbose }}
SKILLSAW_NO_CUSTOM_RULES: ${{ inputs.no-custom-rules }}
SKILLSAW_SINCE: ${{ inputs.since }}
run: |
set +e
ARGS=""
if [ "$SKILLSAW_STRICT" = "true" ]; then
ARGS="$ARGS --strict"
fi
if [ -n "$SKILLSAW_SINCE" ]; then
ARGS="$ARGS --since $SKILLSAW_SINCE"
fi
if [ -n "$SKILLSAW_FAIL_ON" ]; then
case "$SKILLSAW_FAIL_ON" in
error|warning|info) ARGS="$ARGS --fail-on $SKILLSAW_FAIL_ON" ;;
Expand Down
56 changes: 56 additions & 0 deletions docs/baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,62 @@ Run lint without baseline filtering:
skillsaw lint --no-baseline
```

## Lint Only Your Changes (`--since`)

`--since REF` builds the baseline on the fly from git history — no
committed `.skillsaw-baseline.json`, no setup:

```bash
# Report only violations introduced on your branch
skillsaw lint --since origin/main

# Report only violations introduced by the last commit
skillsaw lint --since HEAD~1
```

### How it works

`--since` resolves the **merge-base** of HEAD and REF (the commit your
work diverged from), checks it out into a temporary git worktree, lints
that snapshot with the same configuration, and uses the result as an
ephemeral baseline. The temporary worktree is always removed afterwards.

Because the ephemeral baseline uses the same content-hash fingerprints
as a committed baseline, everything above applies:

- **Drift immunity** — pre-existing violations stay suppressed even when
your change inserts or removes lines around them.
- **Ratchet composition** — value-carrying rules (`context-budget`,
`content-instruction-budget`, `content-actionability-score`) re-fire
only when your change makes the tracked value worse. Growing a
SKILL.md past its merge-base token count reports the regression;
shrinking it stays quiet.
- Fixed violations are reported as
`Baseline: N violation(s) fixed since REF`.

Merge-base semantics mean commits that landed on REF after you branched
are not counted against you — only your own changes are compared.

The rule configuration (`.skillsaw.yaml`, `--rule`/`--skip-rule`,
`--no-custom-rules`, `--no-plugins`) is deliberately taken from the
*current* working tree for both lints: `--since` measures how the
repository changed, not how the rule configuration changed.

`--since` takes precedence over a committed `.skillsaw-baseline.json`
and cannot be combined with `--no-baseline`.

### Limitations

- **Renames resurface old violations** — fingerprints include the
relative file path, so violations in a renamed file no longer match
their merge-base entries and are reported again.
- **Roughly doubles lint time** — the merge-base snapshot is linted in
addition to the working tree.
- **Requires git history** — the merge-base commit must be reachable.
In shallow CI clones, fetch history first (e.g. `fetch-depth: 0` with
`actions/checkout`, or `git fetch --unshallow`); skillsaw reports a
precise error when it cannot resolve the merge-base.

## Stale Entries

When you fix a baselined violation, its baseline entry becomes **stale**.
Expand Down
30 changes: 30 additions & 0 deletions docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ jobs:
| `verbose` | Include info-level violations | `false` |
| `no-custom-rules` | Skip custom rules defined in `.skillsaw.yaml` | `true` |
| `plugins` | Newline-separated list of plugin packages to install | `''` |
| `since` | Only report violations introduced since the merge-base of HEAD and this ref (e.g. `origin/main`) — see [Lint only your changes](baseline.md#lint-only-your-changes-since) | `''` |

### Outputs

Expand All @@ -110,6 +111,35 @@ jobs:
| `warnings` | Number of warnings found |
| `report-file` | Path to JSON report file |

### Lint only the PR's changes

Set `since` to fail CI only on violations the PR itself introduces —
pre-existing violations in the base branch are suppressed automatically,
with no committed baseline file. The merge-base commit must be available,
so check out with full history:

```yaml
jobs:
skillsaw:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
with:
fetch-depth: 0 # --since needs history to find the merge-base
persist-credentials: false
- uses: stbenjam/skillsaw@v0
with:
strict: true
since: origin/${{ github.base_ref }}
```

(Instead of `fetch-depth: 0`, fetching just the base branch also works:
`git fetch origin "$BASE_REF"`.)
Comment on lines +136 to +137

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Define BASE_REF before using it.

BASE_REF is never set in this snippet, so the fallback fetch command fails if someone copies it verbatim. Inline the GitHub expression or define the variable first.

💡 Proposed fix
-(Instead of `fetch-depth: 0`, fetching just the base branch also works:
-`git fetch origin "$BASE_REF"`.)
+(Instead of `fetch-depth: 0`, fetching just the base branch also works:
+`git fetch origin "${{ github.base_ref }}"`.)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
(Instead of `fetch-depth: 0`, fetching just the base branch also works:
`git fetch origin "$BASE_REF"`.)
(Instead of `fetch-depth: 0`, fetching just the base branch also works:
`git fetch origin "${{ github.base_ref }}"`.)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/ci.md` around lines 136 - 137, The fallback fetch example uses BASE_REF
without defining it first, so update the docs snippet to either inline the
GitHub base-branch expression directly in the git fetch command or introduce
BASE_REF before the fetch call. Keep the guidance centered on the ci.md example
so readers can copy it safely without relying on an unset variable.


`since` composes with the review action above: the uploaded report only
contains the PR's own regressions, so inline comments shrink to exactly
what the PR introduced.

### Supply Chain Protection

The examples above use `@v0` for brevity. For supply-chain protection,
Expand Down
1 change: 1 addition & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Lint agent skills, plugins, and AI coding assistant context
| `--rule` | Only run these rules (repeatable). Config still comes from .skillsaw.yaml. | |
| `--skip-rule` | Skip these rules (repeatable). Cannot be combined with --rule. | |
| `--no-baseline` | Ignore baseline file even if .skillsaw-baseline.json exists | |
| `--since` | Only report violations introduced since the merge-base of HEAD and REF (e.g. origin/main). Lints that commit in a temporary git worktree and uses the result as an ephemeral baseline. Takes precedence over .skillsaw-baseline.json; cannot be combined with --no-baseline. Requires git history for the merge-base. | |
| `--no-custom-rules` | Skip custom rules defined in .skillsaw.yaml (recommended for CI on untrusted PRs) | |
| `--no-plugins` | Skip rules from installed plugin packages (skillsaw.plugins entry points) | |
| `--no-progress` | Disable the interactive per-rule progress indicator (auto-disabled when stderr is not a terminal) | |
Expand Down
66 changes: 53 additions & 13 deletions src/skillsaw/cli/_lint.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@ def _run_lint(args):
)
sys.exit(1)

if args.since and args.no_baseline:
print(
"Error: --since and --no-baseline cannot be combined "
"(--since builds its own ephemeral baseline from git)",
file=sys.stderr,
)
sys.exit(1)

output_formats = {}
for spec in args.outputs:
try:
Expand Down Expand Up @@ -111,8 +119,40 @@ def _run_lint(args):
else:
fail_level = config.effective_fail_level()

rule_ids = set(args.rule_ids) if args.rule_ids else None
skip_rule_ids = set(args.skip_rule_ids) if args.skip_rule_ids else None
if rule_ids and skip_rule_ids:
print("Error: --rule and --skip-rule cannot be combined", file=sys.stderr)
sys.exit(1)

baseline = None
if not args.no_baseline:
if args.since:
# --since builds an ephemeral baseline from the merge-base of HEAD
# and the given ref; it takes precedence over any committed
# .skillsaw-baseline.json.
from ..git_baseline import GitBaselineError, build_git_baseline, repo_toplevel

try:
repo_root = repo_toplevel(paths[0])
baseline, merge_base = build_git_baseline(
repo_root,
args.since,
config,
paths,
cli_version,
repo_types=override_types,
rule_ids=rule_ids,
skip_rule_ids=skip_rule_ids,
no_custom_rules=args.no_custom_rules,
no_plugins=args.no_plugins,
)
Comment on lines +137 to +148

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Pass override_types (which holds the --type overrides) to build_git_baseline so that the merge-base snapshot is linted with the correct repository types.

Suggested change
baseline, merge_base = build_git_baseline(
repo_root,
args.since,
config,
paths,
cli_version,
rule_ids=rule_ids,
skip_rule_ids=skip_rule_ids,
no_custom_rules=args.no_custom_rules,
no_plugins=args.no_plugins,
)
baseline, merge_base = build_git_baseline(
repo_root,
args.since,
config,
paths,
cli_version,
rule_ids=rule_ids,
skip_rule_ids=skip_rule_ids,
no_custom_rules=args.no_custom_rules,
no_plugins=args.no_plugins,
repo_types=override_types,
)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in b869de0 — the call site now passes repo_types=override_types so --type applies to the merge-base snapshot lint too. Covered by test_type_override_applies_to_merge_base_lint, which fails without the fix.

except GitBaselineError as e:
print(f"Error: {e}", file=sys.stderr)
sys.exit(1)

if args.fmt == "text":
print(f"Comparing against merge-base {merge_base[:12]} (--since {args.since})\n")
elif not args.no_baseline:
from ..baseline import find_baseline, load_baseline

baseline_path = find_baseline(config.config_dir or paths[0])
Expand All @@ -128,12 +168,6 @@ def _run_lint(args):
except (ValueError, OSError) as e:
print(f"Warning: Failed to load baseline: {e}", file=sys.stderr)

rule_ids = set(args.rule_ids) if args.rule_ids else None
skip_rule_ids = set(args.skip_rule_ids) if args.skip_rule_ids else None
if rule_ids and skip_rule_ids:
print("Error: --rule and --skip-rule cannot be combined", file=sys.stderr)
sys.exit(1)

lint_started = time.perf_counter()
all_violations = []
all_rules = []
Expand Down Expand Up @@ -188,16 +222,22 @@ def _run_lint(args):
if baseline and args.fmt == "text":
stale = linter.stale_baseline_entries
if stale:
print(
f"Baseline: {len(stale)} stale"
f" {'entry' if len(stale) == 1 else 'entries'}"
f" (violations resolved since baseline was set)"
)
if args.since:
print(f"Baseline: {len(stale)} violation(s) fixed since {args.since}")
else:
print(
f"Baseline: {len(stale)} stale"
f" {'entry' if len(stale) == 1 else 'entries'}"
f" (violations resolved since baseline was set)"
)
if args.verbose:
for entry in stale:
location = f" [{entry.file_path}]" if entry.file_path else ""
print(f" - {entry.rule_id}{location}: {entry.message}")
print(" Run `skillsaw baseline` to update.\n")
if args.since:
print()
else:
print(" Run `skillsaw baseline` to update.\n")

merged_context = _build_merged_context(contexts)
unique_rules = _dedup_rules(all_rules)
Expand Down
10 changes: 10 additions & 0 deletions src/skillsaw/cli/_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,16 @@ def _build_parser():
dest="no_baseline",
help="Ignore baseline file even if .skillsaw-baseline.json exists",
)
lint_parser.add_argument(
"--since",
metavar="REF",
default=None,
help="Only report violations introduced since the merge-base of HEAD "
"and REF (e.g. origin/main). Lints that commit in a temporary git "
"worktree and uses the result as an ephemeral baseline. Takes "
"precedence over .skillsaw-baseline.json; cannot be combined with "
"--no-baseline. Requires git history for the merge-base.",
)
lint_parser.add_argument(
"--no-custom-rules",
action="store_true",
Expand Down
Loading