Context
From the README: "goldset builds, versions, and checks scenarios; an external runner replays the turns against your system." No reference runner ships today, so for newcomers the loop is only closed on paper.
Task
Provide examples/run_scenarios.py that:
- reads a scenario bundle,
- replays each turn's
input against a pluggable "system" callable, and
- writes a transcript that
goldset scenarios check accepts.
Ship a deterministic offline stub system (so CI can run it) and document how to swap in an OpenAI-compatible endpoint behind a clearly opt-in path.
Acceptance criteria
Pointers
src/goldset/scenarios/
examples/
Help wanted
Needs a small design decision on the runner interface (how a "system under test" is passed in). Pairs naturally with the transcript-format docs issue.
Context
From the README: "goldset builds, versions, and checks scenarios; an external runner replays the turns against your system." No reference runner ships today, so for newcomers the loop is only closed on paper.
Task
Provide
examples/run_scenarios.pythat:inputagainst a pluggable "system" callable, andgoldset scenarios checkaccepts.Ship a deterministic offline stub system (so CI can run it) and document how to swap in an OpenAI-compatible endpoint behind a clearly opt-in path.
Acceptance criteria
examples/README.mddocuments both modes.Pointers
src/goldset/scenarios/examples/Help wanted
Needs a small design decision on the runner interface (how a "system under test" is passed in). Pairs naturally with the transcript-format docs issue.