Add a skill for reviewing spec compliance

[pellared]

Background

While working on stabilizing Logs in OpenTelemetry Go, I found that AI-assisted reviews can be highly effective for validating an implementation against the OpenTelemetry specification.

The workflow I used was:

1. Identify the specification requirements that need review and create a separate tracking issue for each area, for example:

   • Audit Compliance: Logs Data Model opentelemetry-go#8543

2. Ask an AI agent to compare the implementation with the relevant specification requirements and produce a compliance report for each issue.

3. When the implementation is compliant, close the issue with a summary of the evidence and conclusions, for example:

   • Audit Compliance: Logs API LoggerProvider opentelemetry-go#8540 (comment)

4. When the review identifies gaps or ambiguities, add a comment describing the findings and proposed follow-up work, for example:

   • Audit Compliance: Logs API Logger opentelemetry-go#5679 (comment)

5. Where useful, extend the review by comparing:

   • Implementations in other OpenTelemetry language repositories.
   • Existing implementations of other signals in the same repository, such as the OpenTelemetry Go Trace SDK.
   • The implementation against the intent of the specification, including cases where the specification itself may be incomplete, ambiguous, or incorrect.

6. Produce a final summary containing a table of all reviewed requirements, findings, and recommended follow-up actions.

This process worked well, but it required repeatedly providing similar instructions and manually organizing the results.

Proposal

Create a reusable AI skill that helps maintainers validate whether a given OpenTelemetry implementation complies with the specification.

The skill should guide an AI agent through a consistent, repeatable, and evidence-based compliance review.

Expected capabilities

The skill should be able to:

• Identify the specification sections relevant to a selected signal, component, or feature.
• Extract normative requirements, including MUST, MUST NOT, SHOULD, and other requirement levels.
• Map each requirement to the relevant implementation and test code.
• Classify each requirement as:
  • Compliant.
  • Partially compliant.
  • Non-compliant.
  • Not implemented.
  • Not applicable.
  • Unclear because of specification ambiguity.
• Provide evidence for every conclusion, including links to specification text, source code, tests, and related issues.
• Identify missing or insufficient test coverage.
• Compare behavior with implementations in other OpenTelemetry language repositories when useful.
• Compare behavior with other signals or components in the same implementation.
• Highlight potential problems or ambiguities in the specification rather than automatically treating every discrepancy as an implementation defect.
• Recommend concrete follow-up actions.

The skill should avoid making compliance claims without supporting evidence.

Suggested workflow

A typical invocation could provide:

• The implementation repository.
• The signal, component, or feature to review.
• The relevant specification section or sections.
• Optionally, an existing GitHub issue in which the report should be posted.
• Optionally, reference implementations or related components to compare.

The skill would then:

1. Build a checklist of applicable normative requirements.
2. Locate the corresponding implementation and tests.
3. Evaluate every requirement.
4. Document evidence, findings, uncertainties, and possible specification issues.
5. Produce an issue-level report.
6. Produce or update an aggregate compliance summary.

Suggested output

For each reviewed area, the skill should produce a report containing:

• Scope of the review.
• Specification sections examined.
• Implementation files and tests examined.
• A requirement-by-requirement compliance table.
• Detailed findings.
• Missing tests or implementation gaps.
• Possible specification ambiguities or defects.
• Recommended follow-up issues or pull requests.
• A concise conclusion suitable for closing the tracking issue when no further work is required.

An aggregate report could use a table similar to:

Area	Requirement	Status	Evidence	Finding	Proposed follow-up
Example area	Link to requirement	Compliant / Gap / Unclear	Links to code and tests	Summary	Issue or proposed action

Benefits

A shared skill would:

• Make compliance reviews more consistent across repositories and languages.
• Reduce the effort required to prepare and repeat review instructions.
• Improve traceability between specification requirements, implementation code, and tests.
• Help maintainers identify both implementation gaps and specification problems.
• Provide a repeatable process for stabilizing signals and preparing releases.

Open questions

• How should the skill should treat explanatory or non-normative specification text?
• Should it support creating or updating GitHub issues directly, or only generate review content for maintainers to post?