Search results

Use more industry-standard WER computation

Status: Open.

Feature

#98 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Jul 3, 2026

Should WER computation/STT eval handle transliteration?

STT

Status: Open.

Feature

#97 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Jul 3, 2026

Let users add custom hugging face models for evaluation

Status: Open.

#62 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on May 20, 2026

Add support for more speech providers and open-source models

Status: Open.

#61 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on May 20, 2026

isolate row level errors in STT benchmarking

Status: Open.

#59 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on May 12, 2026

return ttft for STT benchmarks

Status: Open.

#58 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on May 12, 2026

Make openrouter optional and make it possible to run everything by just setting openai api key

Status: Open.

#53 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on May 6, 2026

retry agent connection inference for 2 more times if failed with 500 error

Status: Open.

#46 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Apr 21, 2026

Add support for selecting model for provider for each eval (tts/stt/llm/simulations) by language selected - multiple models per provider

Status: Open.

#45 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Apr 21, 2026

Add Sarvam LLM-WER and LLM-CER support

STT

Status: Open.

Feature

#44 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Apr 21, 2026

Add support for all languages supported by all providers for LLM/tts/stt/simulations (beyond hardcoded languages for llm tests and simulations)

Status: Open.

#43 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Apr 21, 2026

Allow setting LLM model params like thinking, verbosity, temperature etc. for benchmarking/evaluating

Status: Open.

#39 In ARTPARK-SAHAI-ORG/calibrate;

· dalmia opened on Apr 6, 2026