Add an evaluation harness for file selection and content generation

eval/.gitignore

@@ -0,0 +1,3 @@
+# Run outputs are regenerated every run — never commit them.
+# (Python caches are already covered by the repo-root .gitignore.)
+runs/

eval/README.md

@@ -0,0 +1,103 @@
+# code-to-docs evaluation harness
+
+Measures the quality of code-to-docs's two LLM decisions so changes to discovery or
+content generation can be scored instead of eyeballed:
+
+1. **File selection** — given a code diff, which doc files does the pipeline pick? Scored by
+   precision / recall / F1 against a per-case expected set, with the **index** path and the
+   **full-scan** path scored separately.
+2. **Content generation** — is the rewrite faithful and minimal? Scored by deterministic
+   assertions (must / must-not mention, `NO_UPDATE_NEEDED`) plus a no-code-fence-wrapper check.
+
+It runs the **real** pipeline (`find_relevant_files_optimized`, `ask_ai_for_updated_content`) per
+case and scores the output. The scoring engine under `scoring/` is a verbatim vendoring of
+[`opendatahub-io/agent-eval-harness`](https://github.com/opendatahub-io/agent-eval-harness) — see
+`scoring/PROVENANCE.md`.
+
+## Layout
+
+```
+eval/
+├── eval.yaml              # judges + thresholds (the eval config)
+├── execute.py             # runner: real pipeline per case -> outputs in the runs/ layout
+├── prep_index.py          # one-time: build each case's frozen .doc-index/
+├── aggregate.py           # score N samples -> per-case + overall pass-rates
+├── judges/                # selection (P/R/F1), meta (answer-key guard)
+├── scoring/               # vendored agent-eval-harness scorer (do not edit)
+└── dataset/cases/NNN_*/   # diff.patch + docs/ + annotations.yaml (+ frozen .doc-index/)
+```
+
+## Running it
+
+Set an OpenAI-compatible model in the environment (the same vars the action uses):
+
+```bash
+export MODEL_API_BASE="https://api.openai.com/v1"
+export MODEL_API_KEY="sk-..."
+export MODEL_NAME="gpt-4o-mini"
+```
+
+```bash
+# 1. One-time per case: build the frozen index the discovery path consumes
+python eval/prep_index.py --all
+
+# 2. Run the pipeline over every case, N samples each (N>1 handles LLM non-determinism)
+python eval/execute.py --run-id myrun --samples 3
+
+# 3. Aggregate -> per-case and overall pass-rates across the N samples
+python eval/aggregate.py --run-id myrun --samples 3
+```
+
+`aggregate.py` needs no model (it only scores). To score a single sample directly with the vendored
+scorer instead (one run, blended across cases):
+
+```bash
+cd eval
+AGENT_EVAL_RUNS_DIR=runs PYTHONPATH=. python scoring/score.py judges     --run-id myrun-s01 --config eval.yaml
+AGENT_EVAL_RUNS_DIR=runs PYTHONPATH=. python scoring/score.py regression --run-id myrun-s01 --config eval.yaml   # exit 1 on threshold breach
+```
+
+Dependencies: `pyyaml` (scorer) + the `openai` client the action already uses. ASCII-only fixtures
+(the scorer reads `annotations.yaml` with the platform codec; non-ASCII match strings can corrupt
+on Windows — or run with `PYTHONUTF8=1`).
+
+## What it does NOT do (v1)
+
+- No LLM-as-judge — content scoring is deterministic assertions only. The `eval.yaml` schema
+  reserves the slot; gold-reference judging is a follow-up.
+- No CI workflow — this PR is the harness, runnable locally. CI wiring (PR smoke at N=3, nightly at
+  N=10, gated by `score.py regression` vs a committed baseline) is a separate PR.
+
+## Security / trust model
+
+`eval.yaml` is a **trusted input** — the vendored scorer executes it as code:
+
+- judge `check:` snippets run via `exec()` with full builtins,
+- `if:` conditions are `eval()`'d,
+- `module:` judges are imported by name.
+
+So **write access to `eval.yaml` (and the `judges/` modules it names) is effectively
+code-execution trust** during scoring. That's acceptable here — `eval.yaml` is a committed file
+that only changes through PR review — but it's the boundary to be aware of. The scorer under
+`scoring/` is copied byte-identical from agent-eval-harness (see `scoring/PROVENANCE.md`), so this
+is documented rather than patched; restricting the `exec()` namespace would be a change to land
+upstream first.
+
+## Authoring a case
+
+A case is `dataset/cases/NNN_name/` with:
+
+- `diff.patch` — a git-format unified diff (fed directly; no git repo is built).
+- `docs/` — the documentation tree as it stands before the change.
+- `annotations.yaml` — `expected_files` (the selection answer key, `[]` for negatives) and
+  per-file `content` rules (`must_mention`, `must_not_mention`, `expect_no_update`).
+- `.doc-index/` — generated by `prep_index.py`, committed with the case.
+
+**Rule:** a doc folder's name must not equal any changed code-file basename (e.g. don't pair
+`docs/api/` with a diff on `src/api.py`) — the model mashes the two into a bogus path and the case
+measures the collision instead of the model.
+
+## Cost
+
+Roughly 4-5 model calls per case per sample (discovery x2 paths + content gen) plus a one-time
+~2 calls per doc folder at prep. The 10-case corpus at N=3 is ~135 calls; tiny on gpt-4o-mini.

eval/aggregate.py

@@ -0,0 +1,146 @@
+#!/usr/bin/env python3
+"""eval/aggregate.py — the N-sample layer (DECISIONS D-08, D-21).
+
+execute.py --samples N writes run dirs <base>-s01..sNN, each holding every case. This
+scores each sample (reusing the VENDORED scorer's judge dispatch + record loading — no
+reimplementation, D-21) and reports per-(case, judge) PASS-RATES across the N samples,
+which is what tells systematic behavior apart from LLM noise. Flaked samples (D-22) are
+excluded per judge-stage.
+
+Why this and not score.py directly: score.py aggregates across CASES within one run (a
+blended mean). We need per-CASE rates across SAMPLES.
+
+Usage (from eval/):
+    $env:PYTHONPATH = "."
+    python aggregate.py --run-id <base> --samples N
+"""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+EVAL_DIR = Path(__file__).resolve().parent
+sys.path.insert(0, str(EVAL_DIR / "scoring"))   # agent_eval + score
+sys.path.insert(0, str(EVAL_DIR))                # judges.*
+
+from agent_eval.config import EvalConfig   # noqa: E402
+import score                                # noqa: E402
+
+# Which flake key (written by execute.py meta/flakes.json) invalidates which judge.
+_JUDGE_STAGE = {
+    "selection_f1_index": "selection_index",
+    "selection_f1_scan": "selection_scan",
+    "content_assertions": "content:",   # prefix match (any content:<doc>)
+    "no_fence_wrapper": "content:",
+}
+
+
+def _passes(value, judge_name, thresholds):
+    """Per-sample pass for a judge: numeric -> >= min_mean; bool/min_pass_rate -> truthy."""
+    if value is None:
+        return None
+    if isinstance(value, bool):
+        return bool(value)
+    th = thresholds.get(judge_name, {})
+    if "min_mean" in th:
+        return value >= th["min_mean"]
+    return bool(value)
+
+
+def _flaked(flake_keys, judge_name):
+    stage = _JUDGE_STAGE.get(judge_name, "")
+    if not stage:
+        return False
+    if stage.endswith(":"):
+        return any(k.startswith(stage) for k in flake_keys)
+    return stage in flake_keys
+
+
+def _load_meta(case_dir):
+    """Read per-case meta.json (flakes D-22 + filter_drops D-15) directly from the runs tree.
+
+    Read from disk, not record["files"] — `meta/` is intentionally not an eval.yaml output, so
+    the scorer never loads it into the judge record.
+    """
+    p = case_dir / "meta" / "meta.json"
+    if not p.is_file():
+        return set(), []
+    try:
+        data = json.loads(p.read_text(encoding="utf-8"))
+        return set((data.get("flakes") or {}).keys()), list(data.get("filter_drops") or [])
+    except Exception:
+        return set(), []
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--run-id", required=True, help="base run id (sample dirs: <base>-sNN)")
+    ap.add_argument("--samples", type=int, required=True)
+    ap.add_argument("--config", default=str(EVAL_DIR / "eval.yaml"))
+    ap.add_argument("--runs-dir", default=str(EVAL_DIR / "runs"))
+    args = ap.parse_args()
+
+    cfg = EvalConfig.from_yaml(args.config)
+    judges = score.load_judges(cfg, project_root=EVAL_DIR)
+    judge_names = [j[0] for j in judges]
+    thresholds = cfg.thresholds
+    runs_root = Path(args.runs_dir) / "code-to-docs"
+
+    # results[case][judge] = list of per-sample pass (True/False) ; None entries excluded
+    results = {}
+    flake_total = 0
+    drops_total = 0
+    samples_seen = 0
+
+    for s in range(1, args.samples + 1):
+        rid = args.run_id if args.samples == 1 else f"{args.run_id}-s{s:02d}"
+        cases_dir = runs_root / rid / "cases"
+        if not cases_dir.is_dir():
+            print(f"  warn: missing {cases_dir}", file=sys.stderr)
+            continue
+        samples_seen += 1
+        for case_dir in sorted(d for d in cases_dir.iterdir() if d.is_dir()):
+            cid = case_dir.name
+            rec = score.load_case_record(case_dir, cfg, run_id=rid)
+            flake_keys, drops = _load_meta(case_dir)
+            flake_total += len(flake_keys)
+            drops_total += len(drops)
+            for name, scorer, _cond, _jtype, _n in judges:
+                if _flaked(flake_keys, name):
+                    continue  # exclude flaked sample for this judge (D-22)
+                val, _rat = score._normalize_result(scorer(outputs=rec))
+                p = _passes(val, name, thresholds)
+                if p is None:
+                    continue
+                results.setdefault(cid, {}).setdefault(name, []).append(p)
+
+    # ---- report ----
+    print(f"\nAggregate over {samples_seen} sample(s)  "
+          f"(flakes excluded: {flake_total}, discovery filter-drops: {drops_total})\n")
+    overall = {n: [] for n in judge_names}
+    for cid in sorted(results):
+        print(f"case {cid}:")
+        for name in judge_names:
+            passes = results[cid].get(name, [])
+            if not passes:
+                print(f"    {name:22s} -- (no scored samples)")
+                continue
+            rate = sum(passes) / len(passes)
+            overall[name].append(rate)
+            mark = "OK " if rate >= 0.999 else ("!! " if rate == 0 else " ~ ")
+            print(f"    {name:22s} {mark} {sum(passes)}/{len(passes)} pass  ({rate:.0%})")
+        print()
+
+    print("OVERALL (mean per-case pass-rate per judge):")
+    for name in judge_names:
+        rates = overall[name]
+        if not rates:
+            print(f"    {name:22s} -- (no data)")
+            continue
+        m = sum(rates) / len(rates)
+        print(f"    {name:22s} {m:.0%}   across {len(rates)} case(s)")
+
+
+if __name__ == "__main__":
+    main()

eval/dataset/cases/001_add_cli_flag/.doc-index/api.index.md

@@ -0,0 +1,25 @@
+# API Documentation Index
+
+## Overview
+This documentation serves as a comprehensive guide to the `mytool` API, which provides a convenient Python interface for embedding a data processing pipeline within other applications. It introduces the key components of the API, such as the core objects and their basic usage, helping developers understand how to utilize the API effectively.
+
+## Files Summary
+- **overview.md**: This file provides an overview of the `mytool` API, including descriptions of core objects such as `Pipeline` and `Config`, as well as basic usage instructions for integrating the API into user programs.
+
+## Code Changes That Would Require Documentation Updates
+- Addition or removal of core objects such as `Pipeline` and `Config`.
+- Changes to the method signatures or behavior of the `Pipeline.run` method.
+- Modifications to configuration loading mechanisms, such as changes to how `Config` works or what files it can load.
+- Introduction of new functionalities, settings, or parameters within the API.
+- Deprecation of existing features, including outdated methods, attributes, or classes.
+
+## Key Technical Concepts
+- `Pipeline`: A core component that orchestrates the execution of the data processing.
+- `Config`: A configuration handler that resolves settings from a config file or direct input.
+- Method invocation: `Pipeline(cfg).run()`, which demonstrates how to initiate the pipeline with a configuration.
+- Configuration file format: `config.yaml`, indicating the expected format for the configuration settings.
+
+## Related Components
+- **mytool.pipeline**: Module likely containing the implementation of the `Pipeline` class.
+- **mytool.config**: Module likely responsible for loading and managing configurations via the `Config` class.
+- **data processing pipeline**: The underlying system that the API interacts with, which may involve operations defined outside of the `mytool` API.

eval/dataset/cases/001_add_cli_flag/.doc-index/cli.index.md

@@ -0,0 +1,27 @@
+# CLI Documentation Index
+
+## Overview
+This documentation area provides a detailed reference for the command-line interface (CLI) of `mytool`. It aims to assist users in understanding the various options available when executing the `mytool` commands, as well as offering practical examples for effective utilization.
+
+## Files Summary
+- **cli\reference.md**: This file serves as a comprehensive guide to the options available in the CLI for `mytool`, detailing configuration paths, output directories, and special modes like dry runs. It includes examples of how to use the commands.
+
+## Code Changes That Would Require Documentation Updates
+- Addition, removal, or modification of CLI options (e.g., new flags or parameters).
+- Changes in default values for any configuration options provided by the CLI.
+- Alteration of the behavior of existing commands (e.g., how the dry-run feature works).
+- Introduction of new commands or features that enhance or alter functionality.
+- Changes in file paths or structures related to configuration or output directories.
+
+## Key Technical Concepts
+- `mytool`: The main command that users interact with via the CLI.
+- `--config PATH`: Command-line option for specifying the path to the configuration file.
+- `--output DIR`: Command-line option for defining the output directory for generated files.
+- `--dry-run`: A flag that allows users to simulate the execution of the command without writing changes to the filesystem.
+- Configuration files (e.g., `config.yaml`, `prod.yaml`).
+
+## Related Components
+- Configuration system for defining behaviors and parameters for `mytool`.
+- File system structure relevant to the output generation.
+- User interface considerations for CLI interactions.
+- Error handling mechanisms associated with CLI commands.

eval/dataset/cases/001_add_cli_flag/.doc-index/guide.index.md

@@ -0,0 +1,27 @@
+# GUIDE Documentation Index
+
+## Overview
+The "Guide" folder provides comprehensive documentation for getting started with the `mytool` application. It includes installation instructions, initial configuration, and suggestions for subsequent steps to enhance user experience and understanding of the tool. The goal is to simplify the onboarding process for new users and facilitate their first interaction with the software.
+
+## Files Summary
+- **getting-started.md**: This file serves as an introductory guide that outlines the steps necessary to install `mytool`, perform the initial configuration with a YAML file, and run the application for the first time. It also points users to further resources for advanced configuration options.
+
+## Code Changes That Would Require Documentation Updates
+- Changes to the installation process, such as new dependencies or updated installation commands.
+- Modifications to the structure or required fields of the `config.yaml` file.
+- Updates to the default output directory or any changes in the output format produced by `mytool`.
+- Introduction of new features that enhance or modify the first run experience, including changes to available commands.
+- Changes to the settings available in the configuration guide that affect user customization options.
+- Removal or deprecation of existing commands or configuration settings that may lead to confusion for new users.
+
+## Key Technical Concepts
+- **Installation Command**: `pip install mytool`
+- **Configuration File**: `config.yaml`
+- **Default Output Directory**: `build/`
+- **First Run**: Initial execution process of `mytool` after installation.
+- **Settings**: Configuration options available for user customization.
+
+## Related Components
+- **Configuration Guide**: A document that elaborates on the settings available in the `config.yaml` file.
+- **Output Management Module**: Responsible for handling generated outputs from `mytool`.
+- **Installation Management**: Component overseeing the installation dependencies and setup process for `mytool`.

eval/dataset/cases/001_add_cli_flag/.doc-index/manifest.json

@@ -0,0 +1,25 @@
+{
+  "version": "1.0",
+  "created": "2026-06-18T15:28:51.318669",
+  "folders": {
+    "cli": {
+      "built": "2026-06-18T15:28:57.010284",
+      "doc_hashes": {
+        "cli\\reference.md": "6587d14b2647341f8f7bc7eccb362e2e4ae75babce7a7f1b067e7c24e08a56bd"
+      }
+    },
+    "api": {
+      "built": "2026-06-18T15:28:57.650531",
+      "doc_hashes": {
+        "api\\overview.md": "d9776fa54d0883e1e8c540dd3e7fc5e38342d8920a6464b5366778bf2bd8551d"
+      }
+    },
+    "guide": {
+      "built": "2026-06-18T15:28:58.085607",
+      "doc_hashes": {
+        "guide\\getting-started.md": "ab88ad9d4184aaa62404ef26239ac58e924b6c54995ea376fe5dc44e4d453c2c"
+      }
+    }
+  },
+  "updated": "2026-06-18T15:28:58.088606"
+}

eval/dataset/cases/001_add_cli_flag/annotations.yaml

@@ -0,0 +1,13 @@
+# Answer key for case 001. ASCII-only (vendored score.py reads this with the Windows
+# default codec; non-ASCII must_mention strings would corrupt and break matching).
+description: positive - adds --verbose; cli/reference.md should be selected and updated
+# Selection answer key (both discovery paths score against this):
+expected_files:
+  - cli/reference.md
+# Content answer key, per file that content-gen runs on (the gold targets, D-20):
+content:
+  cli/reference.md:
+    must_mention: ["--verbose"]        # the new flag must appear
+    must_not_mention: ["--quiet"]      # a plausible flag NOT in the diff (catches over-eager add)
+    # max_new_lines / must_not_add_sections deferred until execute.py emits the original
+    # baseline (DECISIONS open-questions).

eval/dataset/cases/001_add_cli_flag/diff.patch

@@ -0,0 +1,11 @@
+diff --git a/src/main.py b/src/main.py
+index 7a1b2c3..9d4e5f6 100644
+--- a/src/main.py
++++ b/src/main.py
+@@ -12,6 +12,7 @@ def build_parser():
+     parser = argparse.ArgumentParser(prog="mytool")
+     parser.add_argument("--config", metavar="PATH", help="Path to the configuration file.")
+     parser.add_argument("--output", metavar="DIR", help="Directory for generated output.")
++    parser.add_argument("--verbose", action="store_true", help="Enable verbose logging output.")
+     parser.add_argument("--dry-run", action="store_true", help="Simulate without writing files.")
+     return parser