Skip to content

feat(release-audit-report): add structured audit-record schema and schema-violation eval#722

Merged
potiuk merged 1 commit into
apache:mainfrom
justinmclean:release-audit-report-schema
Jul 4, 2026
Merged

feat(release-audit-report): add structured audit-record schema and schema-violation eval#722
potiuk merged 1 commit into
apache:mainfrom
justinmclean:release-audit-report-schema

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

Add skills/release-audit-report/audit-record-schema.md — a canonical definition of required vs optional audit-record fields, the schema-violation reporting contract, and the privacy boundary for each field. Update release-audit-report/SKILL.md to reference the schema in Step 2, add schema_violations to the Step 2 JSON output contract, and extend the Step 4 hand-back artefact to surface violations to the RM.

Extend the eval suite (step-2-assemble-record) with a has_schema_violations_consistent judge assertion across all cases, update the output-spec to document the new field, add schema_violations to the three existing expected.json files, and add
case-4-all-required-missing — a fixture that proves the skill reports all nine required-field violations when none of the lifecycle evidence was recorded on the planning issue.

Generated-by: Claude (Opus 4.7)

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:

…hema-violation eval

Add `skills/release-audit-report/audit-record-schema.md` — a canonical
definition of required vs optional audit-record fields, the schema-violation
reporting contract, and the privacy boundary for each field.  Update
`release-audit-report/SKILL.md` to reference the schema in Step 2, add
`schema_violations` to the Step 2 JSON output contract, and extend the
Step 4 hand-back artefact to surface violations to the RM.

Extend the eval suite (step-2-assemble-record) with a
`has_schema_violations_consistent` judge assertion across all cases, update
the output-spec to document the new field, add `schema_violations` to the
three existing expected.json files, and add
`case-4-all-required-missing` — a fixture that proves the skill reports
all nine required-field violations when none of the lifecycle evidence was
recorded on the planning issue.

Generated-by: Claude (Opus 4.7)
@justinmclean justinmclean self-assigned this Jul 4, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — behaviour change (Step 2 now emits a schema_violations list) ships with its matching eval: a canonical audit-record-schema.md, a new case-4-all-required-missing fixture, and a has_schema_violations_consistent judge assertion. The fixtures are internally consistent with the schema and the judge rubric. CI green.

Non-blocking nit: the README/case-4 wording says "nine required fields" while audit-record-schema.md lists ten required rows (the tenth, version, is always supplied from the trigger arg so never surfaces as MISSING) — worth tidying the wording on a later pass.

@potiuk
potiuk merged commit 97f25e3 into apache:main Jul 4, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants