Skip to content

Provide a reference scenario runner in examples/ #8

Description

@AshwinUgale

Context

From the README: "goldset builds, versions, and checks scenarios; an external runner replays the turns against your system." No reference runner ships today, so for newcomers the loop is only closed on paper.

Task

Provide examples/run_scenarios.py that:

  1. reads a scenario bundle,
  2. replays each turn's input against a pluggable "system" callable, and
  3. writes a transcript that goldset scenarios check accepts.

Ship a deterministic offline stub system (so CI can run it) and document how to swap in an OpenAI-compatible endpoint behind a clearly opt-in path.

Acceptance criteria

  • Offline stub run is deterministic and exercised by a test.
  • No network dependency in the core or default test path.
  • OpenAI-compatible mode is opt-in and clearly separated (no import at module top level that requires network/keys).
  • examples/README.md documents both modes.

Pointers

  • src/goldset/scenarios/
  • examples/

Help wanted

Needs a small design decision on the runner interface (how a "system under test" is passed in). Pairs naturally with the transcript-format docs issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions