[CI][Bugfix] Compute WER with jiwer directly in transcription tests

[mosya415]

Purpose

test_transcription_api_correctness computes WER through evaluate.load("wer"), which fetches a metric loading script from the Hugging Face Hub whenever that script is not already cached in the CI image. On a cache miss that fetch runs hf_api.HfFolder.get_token(), and with the pinned pair evaluate==0.4.3 and huggingface-hub==1.22.0 it raises AttributeError: module 'huggingface_hub.hf_api' has no attribute 'HfFolder', because HfFolder was removed in huggingface-hub v1. The three WER cases then fail even though the served transcriptions are fine, which makes the Speech to Text job flaky depending on the image cache state.

This was observed in build 80058, job Entrypoints Integration (Speech to Text): https://buildkite.com/vllm/ci/builds/80058#019f957e-48fa-49fd-a234-4bb51424b41e where test_wer_correctness and test_long_audio_wer_correctness failed with that HfFolder error while every non-WER speech case passed.

The fix removes the network path entirely. evaluate's wer metric is a thin wrapper over jiwer, and jiwer==3.0.5 is already a pinned test dependency, so WER is now computed in-process with jiwer.wer(reference=..., hypothesis=...). The numeric result is the same, so the expected WER thresholds in the test are unaffected.

Not a duplicate

No open issue or PR addresses this. I searched vLLM issues and PRs for HfFolder, for wer plus HfFolder, and for open work touching test_transcription_api_correctness, and found only the already merged reliability change #23854, which is orthogonal.

Test Plan

Run the transcription correctness tests, which now compute WER locally with no Hub access:

pytest -v -s tests/entrypoints/speech_to_text/correctness/test_transcription_api_correctness.py

Test Result

Locally verified against jiwer==3.0.5 that jiwer.wer accepts the reference and hypothesis lists this test passes and returns the same fractional WER that evaluate's wrapper produced, so the pinned thresholds stay valid. py_compile and the import-order check pass on the changed file. The full GPU end to end job was not rerun for this test-only change, since it requires the CI hardware and datasets.

Model Evaluation

Not applicable. This changes how a test computes a metric and does not touch model behavior or serving.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mosya415]

cc @NickLucche, small CI fix for the transcription correctness tests, would appreciate a look and the verified/ready labels to run CI when it looks good.

test_transcription_api_correctness computes WER via evaluate.load("wer"), which downloads a metric script from the HF Hub on a cache miss and then calls hf_api.HfFolder.get_token(). With evaluate==0.4.3 against huggingface-hub==1.22.0 that raises AttributeError: module 'huggingface_hub.hf_api' has no attribute 'HfFolder', since HfFolder was removed in huggingface-hub v1, so the WER cases fail whenever the metric is not already cached. Seen in build 80058: https://buildkite.com/vllm/ci/builds/80058#019f957e-48fa-49fd-a234-4bb51424b41e

The fix computes WER with jiwer directly, which is already a pinned test dependency and is what evaluate's wer metric wraps, so there is no network access and the scores are unchanged. Thanks!

[taneem-ibrahim]

This seems to be a duplicate of #49773 . Could you please collaborate on the other PR instead?

[mosya415]

Thanks @taneem-ibrahim, you're right, #49773 predates this and takes the same approach. Closing this one as a duplicate. Happy to review or help over there. Thanks @Change72 for the fix.