Let a host gate on what a run did, not just what it claims - #207
Merged
Conversation
`ready_metadata_validator` receives only model-written metadata, so a host can verify the model's claims about its work but never the work itself. That is enough for "assert nothing you did not support" and not enough for "do not answer without consulting the data" -- a behavioural rule needs what the engine observed, not what the answer says about itself. Add `ready_answer_validator`, called with a `ReadyAnswerState` carrying the metadata plus answer content, iteration, and call counts. Both hooks run when both are supplied and violations from either revoke readiness the same way, so the older hook keeps working unchanged. Trace ABI v8 -> v9: `startup` now reports `ready_gates`, the gates the run actually armed. Enforcement is opt-in and defaults to off, so a host that never passes policy_hints runs with ready_violations and contract_violations permanently inert -- and until now emitted a trace indistinguishable from a fully gated run. One embedder shipped for months in that state while its benchmarks passed hints and measured the enforced configuration; nothing could fail, because nothing recorded which system was running. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ABI is cross-language and the fixture corpus is a shipped artifact, so bumping the Python constant alone left three consumers on v8: the TypeScript relay's envelope check and startup schema, the golden NDJSON the Deno tests compare byte for byte, and the CI step asserting distributions bundle the exact fixture filenames. Teach the relay's startup schema about ready_gates, move every v8 envelope literal and fixture path to v9, and reformat step.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ready_metadata_validatoris handed only model-written metadata. A host can therefore check the model's claims about its work, but never the work — which is enough for "assert nothing you did not support" and not enough for "do not answer without consulting the data".That gap has a concrete cost. A downstream host shipped an answer reading "I don't have enough of your conversation with X available" on a corpus holding 15,333 matching messages, having never called an accessor. Its metadata-only validator saw an answer citing nothing, found nothing to reject, and passed it.
What
ready_answer_validator— optional, receivesReadyAnswerState(metadata, content, iteration, calls made, successful calls). Purely additive;ready_metadata_validatoris unchanged and still supported. Both run when both are supplied, and violations from either revoke readiness identically.startupgains optionalready_gates, naming the gates the run armed.On the ABI bump
Enforcement here is opt-in and defaults to off, so a host that never passes
policy_hintsruns withready_violationsandcontract_violationspermanently inert — and emitted a trace identical to a fully gated run. One embedder shipped months in that state while its benchmark suite passed hints and measured the enforced configuration. No test could fail, because nothing recorded which system was running.That is precisely the silent degradation
UPGRADING.mdexists to document, and it follows the v7 → v8 precedent (that bump was also additive — a newheartbeatevent).Embedders should assert on
ready_gatesin whatever verifies their production config, and treat an unexpectedly empty array as a misconfiguration rather than a default.Verification
1261 passed, 3 skipped. Includes the v8 → v9 fixture/helper rename and a new test pinning the
ready_gatesschema.🤖 Generated with Claude Code