Skip to content

Add confidence signals, human-review gate, no-op removal, and file-list targeting to docs sweeps - #251

Merged
theletterf merged 1 commit into
mainfrom
feat/sweep-confidence-human-gate-target-files
Jul 27, 2026
Merged

Add confidence signals, human-review gate, no-op removal, and file-list targeting to docs sweeps#251
theletterf merged 1 commit into
mainfrom
feat/sweep-confidence-human-gate-target-files

Conversation

@theletterf

Copy link
Copy Markdown
Member

Implements four docs-quality-sweep improvements surfaced by the AI & Docs Challenge cohort (June 2026). New shared fragment gh-aw-fragments/findings-contract.md carries the cross-cutting output contract; all seven sweeps import it. Lock files recompiled with gh-aw v0.83.1 (0 errors, 0 warnings).

Confidence signals — elastic/docs-eng-team#621

  • Adds a per-finding confidence field (high / medium / low) with explicit tier criteria (a separate axis from severity) to every sweep, surfaced in the fix-issue YAML.
  • docs-issue-scope: strengthens the existing Confidence rating so terminology or capabilities that appear only in an issue/PR (this runs with min-integrity: none) are attributed rather than asserted and rated Low, with a caveat line under the table — directly targeting the "wrong terminology propagated into Security docs" incident.

Human-review gate — elastic/docs-eng-team#622

  • When a fix-issue contains any medium/low-confidence finding, the sweep adds a needs-human-review label and a review-before-acting > [!WARNING] callout above the findings.
  • SizeBot (size-logic.md) will not apply good-for-ai to an issue labeled needs-human-review, keeping uncertain output off the auto-delegation track.
  • Intended agent workflow documented in the orchestrator and index READMEs.

Finding-type allowlist + no-op removal — elastic/docs-eng-team#623

  • Removes the missing-h1-anchor category from the openings sweep. Verified against the docs-builder syntax reference: every heading auto-generates a default anchor and an H1 is unique per page, so adding a custom [anchor-id] to an H1 changes nothing rendered — the confirmed no-op from the cohort.
  • Adds a closed category allowlist to every sweep (via the shared contract): no invented/pluralized/hallucinated categories, and drop any finding whose fix wouldn't change rendered output.

File-list targeting — elastic/docs-eng-team#625

  • New target-files input (newline- or comma-separated, docs-root-relative) on the orchestrator and all seven sub-sweeps. When set, it overrides target-path/scope-mode and processes exactly those files — the precise, low-cost path for post-merge "check only what changed" runs. Cost scales with the file count, not directory size.

Testing

  • make compile → 12 workflows, 0 errors / 0 warnings.
  • bash -n on the injected pre-step scripts (shared + typos variant).
  • Functional simulation of the target-files parser against a temp docs tree: comma/newline mixing, leading-slash stripping, and missing-file handling. Caught and fixed a last-entry-dropped bug (while read without trailing newline) — now uses while IFS= read -r raw || [ -n "$raw" ].

Closes elastic/docs-eng-team#621
Closes elastic/docs-eng-team#622
Closes elastic/docs-eng-team#623
Closes elastic/docs-eng-team#625

🤖 Generated with Claude Code

…st targeting to docs sweeps

Implements four docs-quality-sweep improvements from the AI & Docs Challenge cohort (June 2026):

- Confidence signals (elastic/docs-eng-team#621): new shared
  gh-aw-fragments/findings-contract.md adds a per-finding `confidence`
  (high/medium/low) field with explicit tier criteria to all seven sweeps;
  strengthens docs-issue-scope so unverified terminology from low-integrity
  input is attributed rather than asserted and rated Low.
- Human-review gate (elastic/docs-eng-team#622): sweeps conditionally apply a
  `needs-human-review` label and a review-before-acting callout when any finding
  is medium/low confidence; SizeBot no longer applies `good-for-ai` to
  needs-human-review issues; intended agent workflow documented.
- Finding-type allowlist + no-op removal (elastic/docs-eng-team#623): removes the
  `missing-h1-anchor` category from the openings sweep (docs-builder
  auto-generates a heading anchor and H1s are unique, so the fix changes nothing
  rendered); adds a closed category allowlist to every sweep.
- File-list targeting (elastic/docs-eng-team#625): new `target-files` input on the
  orchestrator and all seven sub-sweeps overrides target-path/scope-mode to
  process exactly the listed files.

Recompiled all lock files with gh-aw v0.83.1 (0 errors, 0 warnings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@theletterf
theletterf requested a review from a team as a code owner July 27, 2026 09:33
@theletterf
theletterf requested review from cotti and reakaleek July 27, 2026 09:33
@theletterf theletterf self-assigned this Jul 27, 2026
@theletterf theletterf added the enhancement New feature or request label Jul 27, 2026
@theletterf
theletterf merged commit c38b028 into main Jul 27, 2026
4 of 5 checks passed
@theletterf
theletterf deleted the feat/sweep-confidence-human-gate-target-files branch July 27, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants