Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Governed prompt templates (ISO 42001 A.5.2). Any change to a prompt asset
# requires sign-off from a designated Prompt Engineer / Product Architect.
# See kai_mcp_solution_server/PROMPT_GOVERNANCE.md.
/kai_mcp_solution_server/src/kai_mcp_solution_server/prompts/ @konveyor/kai-prompt-reviewers
/kai_mcp_solution_server/scripts/prompts_version.py @konveyor/kai-prompt-reviewers
48 changes: 48 additions & 0 deletions .github/workflows/prompt-validation.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Prompt Template Validation (ISO 42001 A.5.2)

# Governance gate for the versioned prompt template set in the solution server.
# Runs whenever a governed prompt asset, the manifest, the validation script, the
# render registry, or this workflow changes. Required-status + CODEOWNERS review
# for the prompts directory are documented in PROMPT_GOVERNANCE.md.
on:
pull_request:
paths:
- "kai_mcp_solution_server/src/kai_mcp_solution_server/prompts/**"
- "kai_mcp_solution_server/scripts/prompts_version.py"
- "kai_mcp_solution_server/tests/prompts/**"
- ".github/workflows/prompt-validation.yml"
workflow_dispatch:

jobs:
validate-prompts:
name: Validate prompt templates
runs-on: ubuntu-latest
defaults:
run:
shell: bash
working-directory: ./kai_mcp_solution_server
steps:
- uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install the latest version of uv
uses: astral-sh/setup-uv@v6
with:
version: "latest"

- name: Install dependencies
run: uv sync

# Syntactic verification + version governance: every template declared in
# manifest.yaml (and vice-versa), checksums match, Jinja parses, declared
# variables match the template's referenced variables, no leftover f-strings.
- name: Syntactic & manifest validation
run: uv run python scripts/prompts_version.py check

# Byte-exact parity + deterministic (mock-model) semantic regression.
- name: Parity & semantic-regression tests
run: uv run python -m pytest tests/prompts -v
6 changes: 3 additions & 3 deletions .trunk/trunk.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,13 @@ cli:
plugins:
sources:
- id: trunk
ref: v1.6.3
ref: v1.10.2
uri: https://github.com/trunk-io/plugins
# Many linters and tools depend on runtimes - configure them here. (https://docs.trunk.io/runtimes)
runtimes:
enabled:
- go@1.25.0
- node@18.12.1
- node@22.16.0
- python@3.10.8
# This is the section where you manage your linters. (https://docs.trunk.io/check/configuration)
lint:
Expand All @@ -36,7 +36,7 @@ lint:
- checkov@3.2.521
- git-diff-check
- isort@8.0.1
- markdownlint@0.48.0
- markdownlint@0.49.0
- osv-scanner@2.3.5
- oxipng@10.1.0
- prettier@3.8.3
Expand Down
78 changes: 78 additions & 0 deletions kai_mcp_solution_server/PROMPT_GOVERNANCE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Prompt Template Governance

This document defines the change-management lifecycle for the LLM **prompt templates** the Kai
solution server uses to generate migration hints. It satisfies ISO/IEC 42001 Control **A.5.2
(Governance and lifecycle of AI systems)**: the hint prompt is the instruction set defining the
server's hint-generation behavior, so it is treated as governed source code — version-controlled,
peer-reviewed, and validated by CI before distribution.

## Scope — what is governed

All model-bound prompt templates live as individual Jinja2 assets under
[`src/kai_mcp_solution_server/prompts/templates/`](src/kai_mcp_solution_server/prompts/templates/) and
are enumerated in
[`src/kai_mcp_solution_server/prompts/manifest.yaml`](src/kai_mcp_solution_server/prompts/manifest.yaml).
Today this is the live hint prompt, `generate_hint_v3`. Prompts are rendered by
[Jinja2](https://jinja.palletsprojects.com/) via `render_prompt(prompt_id, **context)` from
`kai_mcp_solution_server.prompts`; loops and structure live in the template, not in `server.py`. The
prompt-set is semantically versioned by `version:` in the manifest, kept in lockstep with the package
version in `pyproject.toml`.

The `ast_diff_str` value (computed from `associate_files` + `extract_ast_info`) is data marshalling
and stays in Python; the template receives it as a variable.

## Roles

- **Prompt Engineer / Product Architect** (`@konveyor/kai-prompt-reviewers`) — required approver for any
change under `prompts/`. Owns prompt wording, the threat-model review, and the regression baseline.
- **Solution-server maintainers** — co-reviewers for the code that calls `render_prompt`.

The prompts directory is assigned to the reviewers in [`.github/CODEOWNERS`](../.github/CODEOWNERS).

## Lifecycle of a prompt change

1. **Branch & edit** the template asset(s). Do not reintroduce prompt strings inline in `server.py`.
2. **Refresh checksums.** Run `python scripts/prompts_version.py update`. Bump `version:` in the
manifest (and `pyproject.toml`) if the change ships in a new release.
3. **Update parity expectations.** The byte-exact oracle (`tests/prompts/oracle.py`) is the historical
baseline. For an _intentional_ wording change, update the oracle and call it out in the PR.
4. **Complete the prompt-injection threat-model checklist** (below) in the PR.
5. **Open a PR** with DCO sign-off (`git commit -s`). CODEOWNERS routes it to a Prompt Engineer; the
[`prompt-validation`](../.github/workflows/prompt-validation.yml) workflow runs automatically.
6. **Merge** only after required review + green CI.

## CI gates

The `prompt-validation` workflow (triggered on `prompts/**`) runs `scripts/prompts_version.py check`
and `pytest tests/prompts`:

- **Syntactic verification** — every template parses as Jinja2; declared variables exactly match the
template's referenced variables; no leftover `{var}` f-string interpolation.
- **Manifest/version governance** — every asset is declared (and vice-versa); content checksums match.
- **Byte-exact parity** — the template reproduces the pre-extraction prompt string exactly across the
incident / AST-diff matrix.
- **Semantic regression (deterministic, mock model)** — renders the prompt against a baseline of
migration scenarios and asserts the required scaffolding (output-format contract, each incident's
fields, the `AST Diff:` anchor) survives.

## Prompt-injection threat-model checklist

Complete for every template change that adds or alters an interpolation point:

- [ ] **Untrusted interpolation** — `incident.message`, `incident.code_snip`, and `ast_diff_str` are
derived from analyzed user code. Confirm they remain data-only and cannot be read by the model
as instructions overriding the output-format contract.
- [ ] **Output-contract integrity** — do the `SUMMARY:` / `HINT:` anchors remain unambiguous so a
malicious code snippet can't spoof the expected response shape?
- [ ] **Rendering** — output is non-escaped by design (these are plain-text prompts); confirm no new
context (JSON, code fence) where unescaped interpolation enables injection.
- [ ] **Regression** — the semantic-regression baseline still passes.

## Branch protection (repo admin action — not in code)

A repository admin must, for `main` and `release-*`:

1. **Require the status check** `Validate prompt templates`.
2. **Require review from Code Owners** so `prompts/**` edits need `@konveyor/kai-prompt-reviewers`.
Comment on lines +75 to +76

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Align required status-check name with the actual workflow/job check.

Line 75 requires Validate prompt templates, but the workflow context is named Prompt Template Validation (ISO 42001 A.5.2). If these don’t match the actual reported check name, branch protection cannot enforce this gate as documented.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@kai_mcp_solution_server/PROMPT_GOVERNANCE.md` around lines 75 - 76, The
required status-check name in PROMPT_GOVERNANCE should match the exact
workflow/job label reported by the CI system. Update the status check entry from
Validate prompt templates to the actual check name used by the prompt validation
workflow, and keep the Code Owners requirement unchanged so the documented
branch protection gate aligns with the status shown by the Prompt Template
Validation (ISO 42001 A.5.2) job.

3. **Create the `@konveyor/kai-prompt-reviewers` team** with the designated Prompt Engineers /
Product Architects.
45 changes: 45 additions & 0 deletions kai_mcp_solution_server/docs/ISO-42001-A5.2-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# ISO/IEC 42001 — Control A.5.2 Evidence Ledger (Kai Solution Server)

**Control:** A.5.2 — Governance and lifecycle of AI systems
**Subject:** Prompt template governance, versioning, and automated validation for the Kai solution
server's hint-generation prompt.
**Status:** Implemented — the prompt is decoupled, version-controlled, peer-review mandated, and
CI-validated.

This is the counterpart to the editor-extensions evidence ledger, which named `konveyor/kai` as the
owner of the solution-server prompt surface. This ledger lists the governed paths.

## 1. Structured versioning & decoupling

| Evidence | Path |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- |
| Dedicated prompt template directory (individual file assets) | [`src/kai_mcp_solution_server/prompts/templates/`](../src/kai_mcp_solution_server/prompts/templates/) |
| Semantic version + per-template manifest with checksums | [`src/kai_mcp_solution_server/prompts/manifest.yaml`](../src/kai_mcp_solution_server/prompts/manifest.yaml) |
| Render registry (template decoupled from app logic) | [`src/kai_mcp_solution_server/prompts/__init__.py`](../src/kai_mcp_solution_server/prompts/__init__.py) |

The live hint prompt (`generate_hint_v3`) is rendered via `render_prompt(...)`; `server.py` no longer
embeds the prompt string. The dead `generate_hint_v1`/`v2` were removed in the same change.

## 2. Automated CI validation pipeline

| Evidence | Path |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Pipeline definition (triggered on `prompts/**`) | [`.github/workflows/prompt-validation.yml`](../../.github/workflows/prompt-validation.yml) |
| Syntactic verification + checksum/version governance | [`scripts/prompts_version.py`](../scripts/prompts_version.py) |
| Byte-exact parity + oracle baseline | [`tests/prompts/test_parity.py`](../tests/prompts/test_parity.py), [`tests/prompts/oracle.py`](../tests/prompts/oracle.py) |
| Semantic regression (deterministic, mock model, baseline dataset) | [`tests/prompts/test_semantic_regression.py`](../tests/prompts/test_semantic_regression.py) |

## 3. Peer review & security policy

| Evidence | Path |
| -------------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| Governance process, roles, lifecycle | [`PROMPT_GOVERNANCE.md`](../PROMPT_GOVERNANCE.md) |
| Mandatory code-owner review for prompt assets | [`.github/CODEOWNERS`](../../.github/CODEOWNERS) |
| Prompt-injection threat-model checklist | [`PROMPT_GOVERNANCE.md`](../PROMPT_GOVERNANCE.md#prompt-injection-threat-model-checklist) |
| Branch-protection requirements (repo-admin action) | [`PROMPT_GOVERNANCE.md`](../PROMPT_GOVERNANCE.md#branch-protection-repo-admin-action--not-in-code) |

## Outstanding (repo-admin / org action)

- Create the `@konveyor/kai-prompt-reviewers` team (Prompt Engineers / Product Architects).
- Enable branch protection on `main` / `release-*` requiring the `Validate prompt templates` check and
Code-Owner review for the prompts directory.
1 change: 1 addition & 0 deletions kai_mcp_solution_server/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ dependencies = [
"tree-sitter>=0.24.0",
"tree-sitter-java>=0.23.5",
"types-pyyaml>=6.0.12.20250516",
"jinja2>=3.1.6",
]

# https://docs.astral.sh/uv/configuration/build-backend/#modules
Expand Down
1 change: 1 addition & 0 deletions kai_mcp_solution_server/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ jaraco-classes==3.4.0
jaraco-context==6.1.0
jaraco-functools==4.4.0
jeepney==0.9.0
jinja2==3.1.6
jiter==0.12.0
jmespath==1.0.1
jsonpatch==1.33
Expand Down
153 changes: 153 additions & 0 deletions kai_mcp_solution_server/scripts/prompts_version.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
#!/usr/bin/env python3
"""Prompt template governance checks (ISO 42001 A.5.2).

Validates the versioned prompt template set under
``src/kai_mcp_solution_server/prompts/``:

- every ``.jinja`` template is declared in ``manifest.yaml`` (and vice-versa)
- content checksums in the manifest match the files on disk (drift detection)
- every template parses as Jinja2 (syntactic verification)
- declared variables exactly match the template's referenced variables, and no
leftover ``{var}`` f-string interpolation escaped into a template asset

Usage:
python scripts/prompts_version.py check # CI gate (non-zero exit on problems)
python scripts/prompts_version.py update # recompute + write checksums
"""

from __future__ import annotations

import hashlib
import re
import sys
import tomllib
from pathlib import Path

import yaml
from jinja2 import Environment, meta

ROOT = Path(__file__).resolve().parent.parent
PROMPTS_DIR = ROOT / "src" / "kai_mcp_solution_server" / "prompts"
TEMPLATES_DIR = PROMPTS_DIR / "templates"
MANIFEST_PATH = PROMPTS_DIR / "manifest.yaml"
PYPROJECT_PATH = ROOT / "pyproject.toml"

# Matches a leftover Python f-string placeholder like ``{incident.uri}`` while
# ignoring Jinja's own ``{{ ... }}`` / ``{% ... %}``.
_FSTRING_RE = re.compile(r"(?<!\{)\{[A-Za-z_][\w.]*\}(?!\})")


def sha256(text: str) -> str:
return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest()


def load_manifest() -> dict:
return yaml.safe_load(MANIFEST_PATH.read_text())


def check() -> int:
manifest = load_manifest()
templates = manifest.get("templates", [])
problems: list[str] = []
env = Environment() # noqa: S701 # nosec B701 - parses templates, never renders

# The prompt-set version must stay in lockstep with the package version so
# release metadata can't drift (see PROMPT_GOVERNANCE.md).
pkg_version = (
tomllib.loads(PYPROJECT_PATH.read_text()).get("project", {}).get("version")
)
if manifest.get("version") != pkg_version:
problems.append(
f"Prompt-set version ({manifest.get('version')}) does not match the package "
f"version in pyproject.toml ({pkg_version}). Keep them in lockstep."
)

on_disk = {p.relative_to(ROOT).as_posix() for p in TEMPLATES_DIR.rglob("*.jinja")}
declared = {t["path"] for t in templates}
for path in on_disk - declared:
problems.append(f"Template on disk is not declared in manifest.yaml: {path}")
for path in declared - on_disk:
problems.append(
f"Manifest declares a template that does not exist on disk: {path}"
)

for entry in templates:
abs_path = ROOT / entry["path"]
if not abs_path.exists():
continue # missing-file already reported
src = abs_path.read_text()

actual = sha256(src)
if entry.get("checksum") != actual:
problems.append(
f"Checksum drift for {entry['id']} ({entry['path']}). "
f"Manifest: {entry.get('checksum') or '<none>'}, actual: {actual}. "
'Run "python scripts/prompts_version.py update" and review the change.'
)

try:
ast = env.parse(src)
except Exception as err: # noqa: BLE001 - report any parse failure
problems.append(
f"Jinja2 parse error in {entry['id']} ({entry['path']}): {err}"
)
continue

if _FSTRING_RE.search(src):
problems.append(
f"Leftover {{...}} f-string interpolation found in {entry['id']} ({entry['path']})."
)

referenced = meta.find_undeclared_variables(ast)
declared_vars = set(entry.get("variables", []))
for missing in declared_vars - referenced:
problems.append(
f'Declared variable "{missing}" is not referenced in {entry["id"]} ({entry["path"]}).'
)
for undeclared in referenced - declared_vars:
problems.append(
f'Variable "{undeclared}" is used in {entry["id"]} ({entry["path"]}) '
"but not declared in manifest.yaml."
)

if problems:
print("Prompt template validation FAILED:\n", file=sys.stderr)
for p in problems:
print(f" - {p}", file=sys.stderr)
print(f"\n{len(problems)} problem(s) found.", file=sys.stderr)
return 1

print(
f"Prompt template validation passed. version={manifest.get('version')}, "
f"templates={len(templates)}."
)
return 0
Comment thread
coderabbitai[bot] marked this conversation as resolved.


def update() -> int:
manifest = load_manifest()
for entry in manifest.get("templates", []):
entry["checksum"] = sha256((ROOT / entry["path"]).read_text())
header = (
"# Prompt template manifest — governed under ISO 42001 A.5.2.\n"
"# Checksums are maintained by scripts/prompts_version.py; do not edit by hand.\n"
)
MANIFEST_PATH.write_text(
header + yaml.safe_dump(manifest, sort_keys=False, width=1000)
)
print(f"Updated checksums for {len(manifest.get('templates', []))} template(s).")
return 0


def main() -> int:
cmd = sys.argv[1] if len(sys.argv) > 1 else ""
if cmd == "check":
return check()
if cmd == "update":
return update()
print("Usage: python scripts/prompts_version.py <check|update>", file=sys.stderr)
return 2


if __name__ == "__main__":
raise SystemExit(main())
Loading
Loading