Prototype a gated self-harness proposal pipeline

[vu1n]

Why

Self-Harness is directionally aligned with pillbox's ghost-self-harness notes: mine failed traces, propose small harness changes, validate them, and keep only improvements. But pillbox's own eval-family notes are the important guardrail: do not build an optimizer until there is measurable headroom and low enough variance.

References:

• https://arxiv.org/abs/2606.09498
• https://github.com/vu1n/pillbox/blob/main/docs/ghost-self-harness.md
• https://github.com/vu1n/pillbox/blob/main/docs/optimization-eval-family.md

Proposal

Create a gated self-harness proposal pipeline after the eval runner and artifact schema exist.

Initial editable surface should be narrow:

• workspace AGENTS.md or equivalent instruction file
• tool output formatting
• runtime policy toggles
• optional subagent/tool availability

Loop:

1. Mine failed traces for recurring weakness patterns.
2. Generate minimal proposal manifests tied to specific failures.
3. Validate on held-in and held-out tasks.
4. Reject proposals that fail regression, variance, or complexity gates.
5. Persist accepted proposals with source traces and measured deltas.

Non-goals

• No autonomous mutation of production defaults without explicit acceptance.
• No broad prompt rewrites without attribution to a failure class.
• No memory/skill soup before we can measure which surface caused the gain.

Acceptance criteria

• Proposal manifests have a stable schema.
• Every proposal links to source traces and expected failure mode.
• Validation reports include held-in/held-out scores and variance notes.
• Rejected proposals are retained as artifacts or run notes, not silently discarded.
• The pipeline can be run experimentally without changing default pillbox behavior.