Skip to content

feat: add benchmark pull command and dataset-aware manifest sync #30

Description

@kaaloo

Problem

The HF dataset provider (issue #28) exposes granular huggingface dataset {ls,pull,push} subcommands, but a typical user wants a single command that fills both data/audio/ and data/ground_truth/ and refreshes data/manifest.md in one go. Today, manifest sync only walks the local filesystem and never records the origin of a sample, so it is impossible to tell which samples came from a remote dataset and at which revision.

This is the aggregator that makes the new HF dataset store ergonomic for a local user, and the building block the upcoming results-ledger issue (results push) needs to know what corpus revision a given run was scored against.

Scope

  • Add eval-transcript benchmark pull that performs huggingface dataset pull + manifest sync in one step.
  • Make manifest sync aware of the HF dataset origin: each sample's rendered frontmatter records source: huggingface and corpus_revision when those are set, with no change to current behavior when the field is absent.
  • The aggregator does not introduce new flags; it just composes the existing huggingface dataset pull and manifest sync flows.

Subcommands

  • eval-transcript benchmark pull [--samples a,b,c] [--all] [--force] [--revision REV] — equivalent to huggingface dataset pull <args> followed by manifest sync. Prints a single summary line: pulled N, refreshed manifest <path>.
  • eval-transcript manifest sync [--corpus-repo REPO] [--corpus-revision REV] [--source huggingface|local] — when --source is huggingface, records source: huggingface and corpus_revision: REV in the rendered manifest frontmatter per sample. Defaults to the existing local-only behavior.

Acceptance criteria

  • benchmark pull --samples a,b is equivalent to huggingface dataset pull --samples a,b followed by manifest sync, and prints a single summary.
  • benchmark pull (no args) pulls the full default sample list from .env (HF_DATASET_REPO).
  • manifest sync records source: huggingface and corpus_revision: REV per sample when those are set; behavior is unchanged when neither flag is passed.
  • Manifest entries rendered from a non-HF source keep the current shape (no source key).
  • Tests use a tmpdir with a fake HF API. The fake returns two sample rows; benchmark pull writes the audio and ground truth files and renders a manifest with source: huggingface per sample.
  • uv run pre-commit run --all-files is clean.
  • uv run python -m unittest discover -s tests -v is clean.

Constraints

  • The aggregator lives in eval_transcript/__init__.py (CLI wiring) and reuses existing helpers in huggingface.py and manifest.py. No new business logic.
  • manifest sync must remain backward compatible: existing manifests with no source key still render correctly.
  • The corpus_revision is recorded verbatim from the value passed to manifest sync. The aggregator pulls the revision from huggingface dataset pull --revision.

Out of scope

  • Results ledger.
  • Auto-pinning the corpus revision in scoring output (the upcoming results-ledger issue will read the manifest's corpus_revision).

Depends on

👾 Generated with Letta Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions