-
Notifications
You must be signed in to change notification settings - Fork 54
Govern the solution server hint prompt as a versioned Jinja2 template #930
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
fabianvf
wants to merge
3
commits into
konveyor:main
Choose a base branch
from
fabianvf:prompt-template-governance
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+642
−194
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Governed prompt templates (ISO 42001 A.5.2). Any change to a prompt asset | ||
| # requires sign-off from a designated Prompt Engineer / Product Architect. | ||
| # See kai_mcp_solution_server/PROMPT_GOVERNANCE.md. | ||
| /kai_mcp_solution_server/src/kai_mcp_solution_server/prompts/ @konveyor/kai-prompt-reviewers | ||
| /kai_mcp_solution_server/scripts/prompts_version.py @konveyor/kai-prompt-reviewers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| name: Prompt Template Validation (ISO 42001 A.5.2) | ||
|
|
||
| # Governance gate for the versioned prompt template set in the solution server. | ||
| # Runs whenever a governed prompt asset, the manifest, the validation script, the | ||
| # render registry, or this workflow changes. Required-status + CODEOWNERS review | ||
| # for the prompts directory are documented in PROMPT_GOVERNANCE.md. | ||
| on: | ||
| pull_request: | ||
| paths: | ||
| - "kai_mcp_solution_server/src/kai_mcp_solution_server/prompts/**" | ||
| - "kai_mcp_solution_server/scripts/prompts_version.py" | ||
| - "kai_mcp_solution_server/tests/prompts/**" | ||
| - ".github/workflows/prompt-validation.yml" | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| validate-prompts: | ||
| name: Validate prompt templates | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| working-directory: ./kai_mcp_solution_server | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Install the latest version of uv | ||
| uses: astral-sh/setup-uv@v6 | ||
| with: | ||
| version: "latest" | ||
|
|
||
| - name: Install dependencies | ||
| run: uv sync | ||
|
|
||
| # Syntactic verification + version governance: every template declared in | ||
| # manifest.yaml (and vice-versa), checksums match, Jinja parses, declared | ||
| # variables match the template's referenced variables, no leftover f-strings. | ||
| - name: Syntactic & manifest validation | ||
| run: uv run python scripts/prompts_version.py check | ||
|
|
||
| # Byte-exact parity + deterministic (mock-model) semantic regression. | ||
| - name: Parity & semantic-regression tests | ||
| run: uv run python -m pytest tests/prompts -v |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| # Prompt Template Governance | ||
|
|
||
| This document defines the change-management lifecycle for the LLM **prompt templates** the Kai | ||
| solution server uses to generate migration hints. It satisfies ISO/IEC 42001 Control **A.5.2 | ||
| (Governance and lifecycle of AI systems)**: the hint prompt is the instruction set defining the | ||
| server's hint-generation behavior, so it is treated as governed source code — version-controlled, | ||
| peer-reviewed, and validated by CI before distribution. | ||
|
|
||
| ## Scope — what is governed | ||
|
|
||
| All model-bound prompt templates live as individual Jinja2 assets under | ||
| [`src/kai_mcp_solution_server/prompts/templates/`](src/kai_mcp_solution_server/prompts/templates/) and | ||
| are enumerated in | ||
| [`src/kai_mcp_solution_server/prompts/manifest.yaml`](src/kai_mcp_solution_server/prompts/manifest.yaml). | ||
| Today this is the live hint prompt, `generate_hint_v3`. Prompts are rendered by | ||
| [Jinja2](https://jinja.palletsprojects.com/) via `render_prompt(prompt_id, **context)` from | ||
| `kai_mcp_solution_server.prompts`; loops and structure live in the template, not in `server.py`. The | ||
| prompt-set is semantically versioned by `version:` in the manifest, kept in lockstep with the package | ||
| version in `pyproject.toml`. | ||
|
|
||
| The `ast_diff_str` value (computed from `associate_files` + `extract_ast_info`) is data marshalling | ||
| and stays in Python; the template receives it as a variable. | ||
|
|
||
| ## Roles | ||
|
|
||
| - **Prompt Engineer / Product Architect** (`@konveyor/kai-prompt-reviewers`) — required approver for any | ||
| change under `prompts/`. Owns prompt wording, the threat-model review, and the regression baseline. | ||
| - **Solution-server maintainers** — co-reviewers for the code that calls `render_prompt`. | ||
|
|
||
| The prompts directory is assigned to the reviewers in [`.github/CODEOWNERS`](../.github/CODEOWNERS). | ||
|
|
||
| ## Lifecycle of a prompt change | ||
|
|
||
| 1. **Branch & edit** the template asset(s). Do not reintroduce prompt strings inline in `server.py`. | ||
| 2. **Refresh checksums.** Run `python scripts/prompts_version.py update`. Bump `version:` in the | ||
| manifest (and `pyproject.toml`) if the change ships in a new release. | ||
| 3. **Update parity expectations.** The byte-exact oracle (`tests/prompts/oracle.py`) is the historical | ||
| baseline. For an _intentional_ wording change, update the oracle and call it out in the PR. | ||
| 4. **Complete the prompt-injection threat-model checklist** (below) in the PR. | ||
| 5. **Open a PR** with DCO sign-off (`git commit -s`). CODEOWNERS routes it to a Prompt Engineer; the | ||
| [`prompt-validation`](../.github/workflows/prompt-validation.yml) workflow runs automatically. | ||
| 6. **Merge** only after required review + green CI. | ||
|
|
||
| ## CI gates | ||
|
|
||
| The `prompt-validation` workflow (triggered on `prompts/**`) runs `scripts/prompts_version.py check` | ||
| and `pytest tests/prompts`: | ||
|
|
||
| - **Syntactic verification** — every template parses as Jinja2; declared variables exactly match the | ||
| template's referenced variables; no leftover `{var}` f-string interpolation. | ||
| - **Manifest/version governance** — every asset is declared (and vice-versa); content checksums match. | ||
| - **Byte-exact parity** — the template reproduces the pre-extraction prompt string exactly across the | ||
| incident / AST-diff matrix. | ||
| - **Semantic regression (deterministic, mock model)** — renders the prompt against a baseline of | ||
| migration scenarios and asserts the required scaffolding (output-format contract, each incident's | ||
| fields, the `AST Diff:` anchor) survives. | ||
|
|
||
| ## Prompt-injection threat-model checklist | ||
|
|
||
| Complete for every template change that adds or alters an interpolation point: | ||
|
|
||
| - [ ] **Untrusted interpolation** — `incident.message`, `incident.code_snip`, and `ast_diff_str` are | ||
| derived from analyzed user code. Confirm they remain data-only and cannot be read by the model | ||
| as instructions overriding the output-format contract. | ||
| - [ ] **Output-contract integrity** — do the `SUMMARY:` / `HINT:` anchors remain unambiguous so a | ||
| malicious code snippet can't spoof the expected response shape? | ||
| - [ ] **Rendering** — output is non-escaped by design (these are plain-text prompts); confirm no new | ||
| context (JSON, code fence) where unescaped interpolation enables injection. | ||
| - [ ] **Regression** — the semantic-regression baseline still passes. | ||
|
|
||
| ## Branch protection (repo admin action — not in code) | ||
|
|
||
| A repository admin must, for `main` and `release-*`: | ||
|
|
||
| 1. **Require the status check** `Validate prompt templates`. | ||
| 2. **Require review from Code Owners** so `prompts/**` edits need `@konveyor/kai-prompt-reviewers`. | ||
| 3. **Create the `@konveyor/kai-prompt-reviewers` team** with the designated Prompt Engineers / | ||
| Product Architects. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # ISO/IEC 42001 — Control A.5.2 Evidence Ledger (Kai Solution Server) | ||
|
|
||
| **Control:** A.5.2 — Governance and lifecycle of AI systems | ||
| **Subject:** Prompt template governance, versioning, and automated validation for the Kai solution | ||
| server's hint-generation prompt. | ||
| **Status:** Implemented — the prompt is decoupled, version-controlled, peer-review mandated, and | ||
| CI-validated. | ||
|
|
||
| This is the counterpart to the editor-extensions evidence ledger, which named `konveyor/kai` as the | ||
| owner of the solution-server prompt surface. This ledger lists the governed paths. | ||
|
|
||
| ## 1. Structured versioning & decoupling | ||
|
|
||
| | Evidence | Path | | ||
| | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | ||
| | Dedicated prompt template directory (individual file assets) | [`src/kai_mcp_solution_server/prompts/templates/`](../src/kai_mcp_solution_server/prompts/templates/) | | ||
| | Semantic version + per-template manifest with checksums | [`src/kai_mcp_solution_server/prompts/manifest.yaml`](../src/kai_mcp_solution_server/prompts/manifest.yaml) | | ||
| | Render registry (template decoupled from app logic) | [`src/kai_mcp_solution_server/prompts/__init__.py`](../src/kai_mcp_solution_server/prompts/__init__.py) | | ||
|
|
||
| The live hint prompt (`generate_hint_v3`) is rendered via `render_prompt(...)`; `server.py` no longer | ||
| embeds the prompt string. The dead `generate_hint_v1`/`v2` were removed in the same change. | ||
|
|
||
| ## 2. Automated CI validation pipeline | ||
|
|
||
| | Evidence | Path | | ||
| | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | ||
| | Pipeline definition (triggered on `prompts/**`) | [`.github/workflows/prompt-validation.yml`](../../.github/workflows/prompt-validation.yml) | | ||
| | Syntactic verification + checksum/version governance | [`scripts/prompts_version.py`](../scripts/prompts_version.py) | | ||
| | Byte-exact parity + oracle baseline | [`tests/prompts/test_parity.py`](../tests/prompts/test_parity.py), [`tests/prompts/oracle.py`](../tests/prompts/oracle.py) | | ||
| | Semantic regression (deterministic, mock model, baseline dataset) | [`tests/prompts/test_semantic_regression.py`](../tests/prompts/test_semantic_regression.py) | | ||
|
|
||
| ## 3. Peer review & security policy | ||
|
|
||
| | Evidence | Path | | ||
| | -------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | ||
| | Governance process, roles, lifecycle | [`PROMPT_GOVERNANCE.md`](../PROMPT_GOVERNANCE.md) | | ||
| | Mandatory code-owner review for prompt assets | [`.github/CODEOWNERS`](../../.github/CODEOWNERS) | | ||
| | Prompt-injection threat-model checklist | [`PROMPT_GOVERNANCE.md`](../PROMPT_GOVERNANCE.md#prompt-injection-threat-model-checklist) | | ||
| | Branch-protection requirements (repo-admin action) | [`PROMPT_GOVERNANCE.md`](../PROMPT_GOVERNANCE.md#branch-protection-repo-admin-action--not-in-code) | | ||
|
|
||
| ## Outstanding (repo-admin / org action) | ||
|
|
||
| - Create the `@konveyor/kai-prompt-reviewers` team (Prompt Engineers / Product Architects). | ||
| - Enable branch protection on `main` / `release-*` requiring the `Validate prompt templates` check and | ||
| Code-Owner review for the prompts directory. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,153 @@ | ||
| #!/usr/bin/env python3 | ||
| """Prompt template governance checks (ISO 42001 A.5.2). | ||
|
|
||
| Validates the versioned prompt template set under | ||
| ``src/kai_mcp_solution_server/prompts/``: | ||
|
|
||
| - every ``.jinja`` template is declared in ``manifest.yaml`` (and vice-versa) | ||
| - content checksums in the manifest match the files on disk (drift detection) | ||
| - every template parses as Jinja2 (syntactic verification) | ||
| - declared variables exactly match the template's referenced variables, and no | ||
| leftover ``{var}`` f-string interpolation escaped into a template asset | ||
|
|
||
| Usage: | ||
| python scripts/prompts_version.py check # CI gate (non-zero exit on problems) | ||
| python scripts/prompts_version.py update # recompute + write checksums | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import hashlib | ||
| import re | ||
| import sys | ||
| import tomllib | ||
| from pathlib import Path | ||
|
|
||
| import yaml | ||
| from jinja2 import Environment, meta | ||
|
|
||
| ROOT = Path(__file__).resolve().parent.parent | ||
| PROMPTS_DIR = ROOT / "src" / "kai_mcp_solution_server" / "prompts" | ||
| TEMPLATES_DIR = PROMPTS_DIR / "templates" | ||
| MANIFEST_PATH = PROMPTS_DIR / "manifest.yaml" | ||
| PYPROJECT_PATH = ROOT / "pyproject.toml" | ||
|
|
||
| # Matches a leftover Python f-string placeholder like ``{incident.uri}`` while | ||
| # ignoring Jinja's own ``{{ ... }}`` / ``{% ... %}``. | ||
| _FSTRING_RE = re.compile(r"(?<!\{)\{[A-Za-z_][\w.]*\}(?!\})") | ||
|
|
||
|
|
||
| def sha256(text: str) -> str: | ||
| return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest() | ||
|
|
||
|
|
||
| def load_manifest() -> dict: | ||
| return yaml.safe_load(MANIFEST_PATH.read_text()) | ||
|
|
||
|
|
||
| def check() -> int: | ||
| manifest = load_manifest() | ||
| templates = manifest.get("templates", []) | ||
| problems: list[str] = [] | ||
| env = Environment() # noqa: S701 # nosec B701 - parses templates, never renders | ||
|
|
||
| # The prompt-set version must stay in lockstep with the package version so | ||
| # release metadata can't drift (see PROMPT_GOVERNANCE.md). | ||
| pkg_version = ( | ||
| tomllib.loads(PYPROJECT_PATH.read_text()).get("project", {}).get("version") | ||
| ) | ||
| if manifest.get("version") != pkg_version: | ||
| problems.append( | ||
| f"Prompt-set version ({manifest.get('version')}) does not match the package " | ||
| f"version in pyproject.toml ({pkg_version}). Keep them in lockstep." | ||
| ) | ||
|
|
||
| on_disk = {p.relative_to(ROOT).as_posix() for p in TEMPLATES_DIR.rglob("*.jinja")} | ||
| declared = {t["path"] for t in templates} | ||
| for path in on_disk - declared: | ||
| problems.append(f"Template on disk is not declared in manifest.yaml: {path}") | ||
| for path in declared - on_disk: | ||
| problems.append( | ||
| f"Manifest declares a template that does not exist on disk: {path}" | ||
| ) | ||
|
|
||
| for entry in templates: | ||
| abs_path = ROOT / entry["path"] | ||
| if not abs_path.exists(): | ||
| continue # missing-file already reported | ||
| src = abs_path.read_text() | ||
|
|
||
| actual = sha256(src) | ||
| if entry.get("checksum") != actual: | ||
| problems.append( | ||
| f"Checksum drift for {entry['id']} ({entry['path']}). " | ||
| f"Manifest: {entry.get('checksum') or '<none>'}, actual: {actual}. " | ||
| 'Run "python scripts/prompts_version.py update" and review the change.' | ||
| ) | ||
|
|
||
| try: | ||
| ast = env.parse(src) | ||
| except Exception as err: # noqa: BLE001 - report any parse failure | ||
| problems.append( | ||
| f"Jinja2 parse error in {entry['id']} ({entry['path']}): {err}" | ||
| ) | ||
| continue | ||
|
|
||
| if _FSTRING_RE.search(src): | ||
| problems.append( | ||
| f"Leftover {{...}} f-string interpolation found in {entry['id']} ({entry['path']})." | ||
| ) | ||
|
|
||
| referenced = meta.find_undeclared_variables(ast) | ||
| declared_vars = set(entry.get("variables", [])) | ||
| for missing in declared_vars - referenced: | ||
| problems.append( | ||
| f'Declared variable "{missing}" is not referenced in {entry["id"]} ({entry["path"]}).' | ||
| ) | ||
| for undeclared in referenced - declared_vars: | ||
| problems.append( | ||
| f'Variable "{undeclared}" is used in {entry["id"]} ({entry["path"]}) ' | ||
| "but not declared in manifest.yaml." | ||
| ) | ||
|
|
||
| if problems: | ||
| print("Prompt template validation FAILED:\n", file=sys.stderr) | ||
| for p in problems: | ||
| print(f" - {p}", file=sys.stderr) | ||
| print(f"\n{len(problems)} problem(s) found.", file=sys.stderr) | ||
| return 1 | ||
|
|
||
| print( | ||
| f"Prompt template validation passed. version={manifest.get('version')}, " | ||
| f"templates={len(templates)}." | ||
| ) | ||
| return 0 | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
|
|
||
| def update() -> int: | ||
| manifest = load_manifest() | ||
| for entry in manifest.get("templates", []): | ||
| entry["checksum"] = sha256((ROOT / entry["path"]).read_text()) | ||
| header = ( | ||
| "# Prompt template manifest — governed under ISO 42001 A.5.2.\n" | ||
| "# Checksums are maintained by scripts/prompts_version.py; do not edit by hand.\n" | ||
| ) | ||
| MANIFEST_PATH.write_text( | ||
| header + yaml.safe_dump(manifest, sort_keys=False, width=1000) | ||
| ) | ||
| print(f"Updated checksums for {len(manifest.get('templates', []))} template(s).") | ||
| return 0 | ||
|
|
||
|
|
||
| def main() -> int: | ||
| cmd = sys.argv[1] if len(sys.argv) > 1 else "" | ||
| if cmd == "check": | ||
| return check() | ||
| if cmd == "update": | ||
| return update() | ||
| print("Usage: python scripts/prompts_version.py <check|update>", file=sys.stderr) | ||
| return 2 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Align required status-check name with the actual workflow/job check.
Line 75 requires
Validate prompt templates, but the workflow context is namedPrompt Template Validation (ISO 42001 A.5.2). If these don’t match the actual reported check name, branch protection cannot enforce this gate as documented.🤖 Prompt for AI Agents