docs: agent-judge OSS eval adapter — planning (refs promptlm-evals#44)

[fabapp2]

Planning-only PR (draft, no code). Re-rooted from promptlm-evals#45 after we hit the BSL/Apache-2.0 boundary — OSS-tier planning belongs here, in this Apache-2.0 repo.

What's in this PR

Three documents, intended to be argued before any module scaffolding lands:

• docs/proposals/oss-eval-adapter-tier.md

  • Establishes components/promptlm-evaluation-<vendor>/ as the home for third-party evaluation adapters in this repo.
  • Introduces a community vs core tier signalled by a Maven property (<promptlm.eval.tier>community</promptlm.eval.tier>), README badge, and a repo-root docs/eval-adapters.md manifest — not by directory location or build profile.
  • Defines the extensions["x-evaluation-<adapter>"] convention so structured payloads (e.g. per-Check cascade results) can ride alongside the v1 EvaluationResult POJO and bridge cleanly into the commercial v2 schema when promptlm-evals is on the classpath.

• docs/proposals/agent-judge-oss-adapter.md

  • Concrete design for the first adapter under this scheme.
  • Implements host dev.promptlm.domain.promptspec.Evaluation.
  • Two-write mapping: v1 EvaluationResult (score + reasoning summary) and structured Judgment payload to extensions["x-evaluation-agent-judge"] — preserves the per-Check granularity that the Pollack post argues is load-bearing (Kamoi et al., He et al., Zhuo et al.).
  • @ConditionalOnClass(ai.pollack.agentjudge.Judge) auto-config; opt-in via classpath presence.
  • Declarative application.yaml jury composition for users who don't want to author beans.
  • GraalVM native-image scoped as a second milestone, not blocking this one.

• docs/spikes/2026-05-agent-judge.md

  • 2-day time-boxed spike: hard license / Maven Central pre-flight gates, JVM end-to-end checklist mirroring the Pollack cascaded-jury example, payload-shape stress tests (including a mock commercial-bridge read), GraalVM scouting pass, filled-at-end decision matrix → adopt / integrate-via-adapter / inspire.

Why two writes (the structurally interesting decision)

The OSS v1 EvaluationResult POJO is deliberately narrow (evaluator, type, score, reasoning, comments). The whole reason agent-judge is worth integrating is the structured per-Check output — collapsing it to a single score throws away the point. We considered four options:

1. JSON in comments — semantically wrong; tooling that displays comments gets garbage.
2. Add extensions to v1 POJO — pulls v1 toward v2, re-fragments the schema landscape.
3. OSS collapses, structure only via commercial bridge — defeats the value prop for OSS-only users.
4. Extension slot on PromptSpec ✅ — reuses the existing D-33 two-slot persistence pattern. OSS users get the data; commercial bridge lifts it losslessly.

Option 4 is the one this PR adopts. It's the smallest move that honours both the Pollack thesis and the OSS+commercial compatibility goal.

Why a draft PR

The tier structure (oss-eval-adapter-tier.md) and the extension-slot convention are the highest-cost decisions in here. Once they land, the rest follows. PR-as-discussion-thread keeps the argument in one place.

What this PR is not

• No new Maven modules.
• No agent-judge dependency added anywhere.
• No code.
• No docs/eval-adapters.md manifest yet (created when the first adapter actually lands).
• No license verification yet — that is the first spike step and a hard gate.

Out of scope / follow-ups

• GraalVM native-image milestone for agent-judge (file after spike).
• Commercial bridge in promptlm-evals reading extensions["x-evaluation-agent-judge"] and lifting to v2 — to land after the OSS adapter ships, not before.
• Reaching out to the upstream author re: roadmap alignment.

Suggested review order

1. oss-eval-adapter-tier.md (the structural calls — tier model + extension-slot convention).
2. agent-judge-oss-adapter.md (concrete design — only meaningful if the structural calls land).
3. Spike plan (the de-risking step before any module code).

---

🤖 Generated with Claude Code