Skip to content

chore: add CI workflow for end-to-end HF pull/score/push #32

Description

@kaaloo

Problem

The new HF-backed benchmark loop (corpus pull → transcribe → score → results push) should be exercisable from GitHub Actions on a small public corpus so future regressions in the dataset provider, the manifest sync flow, and the results ledger are caught early. Today, the test suite is unit-only and the live HF APIs are never exercised in CI.

Scope

Add a single GitHub Actions workflow, .github/workflows/ci-hf-benchmark.yml, that performs the full loop on a throwaway test org/repo. The workflow runs:

  1. Install uv + the package.
  2. Use a repo-level HF_TOKEN secret bound to a throwaway test org (no real benchmark data).
  3. Pull two fixed samples from a public test corpus (the same org's eval-transcript-corpus-test repo).
  4. Score pre-existing local transcription stubs (no live ASR calls in CI to keep the loop deterministic and free of API costs).
  5. Push a result row to eval-transcript-results-test.
  6. Assert that the result row is queryable after the run.

The workflow uses workflow_dispatch to avoid surprise runs on every PR; it triggers on PRs that touch src/eval_transcript/huggingface.py, src/eval_transcript/results.py, src/eval_transcript/manifest.py, or the workflow file itself.

Acceptance criteria

  • Workflow file .github/workflows/ci-hf-benchmark.yml exists and parses.
  • workflow_dispatch runs the full loop end to end.
  • PRs that touch the listed source files trigger the workflow automatically.
  • The workflow fails on any uncaught exception in the round trip (e.g. missing repo, wrong visibility, schema mismatch).
  • A README "Continuous integration" section documents the workflow and the test org convention.
  • uv run pre-commit run --all-files is clean on the workflow file (.github/**/*.yml is in scope).

Constraints

  • Do not commit any real HF_TOKEN to the repo. The workflow reads HF_TOKEN_CI from GitHub secrets.
  • The throwaway test org/repo names are documented in the workflow comments; the workflow does not assume they exist (it creates them on first run).
  • Live ASR calls are out of scope: the workflow uses pre-existing local transcription stubs in the test org's corpus to keep the loop deterministic and cost-free.
  • The workflow does not have write access to main or any production branch.

Out of scope

  • Nightly full-corpus benchmarks.
  • Provider comparison dashboards consuming the results dataset.
  • Live ASR coverage in CI (still flaky and billed).

Depends on

👾 Generated with Letta Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions