Problem
async-openai 0.36's default Client<OpenAIConfig> ships with ExponentialBackoff retry on all server errors (5xx, 429). The default configuration retries up to 8 times with growing backoff, totaling roughly 15 minutes wall-clock before surfacing the error.
For an interactive transcription use case ("user clicked Transcribe and is staring at the spinner"), 15 minutes of silent retry is the wrong default. The user has no signal that anything is wrong; they don't know whether to wait, retry, or switch providers.
This came up in T4 implementation: a wiremock end-to-end test for the 5xx → BackendUnavailable mapping took ~80s before the test runner even got the chance to assert. We worked around it by replacing the 5xx test with a 200-malformed-body test (JSONDeserialize → Internal), but the production behavior remains as-is.
Suggested approach
Either:
- Disable retry entirely for the transcription client. Unit / integration tests no longer wait. Production users see the error immediately, can decide what to do.
- Shorten retry window to ~30s total (e.g. 2 attempts with 5s + 10s backoff). Compromise — covers transient 502s without keeping the user waiting through a real outage.
- Surface retry progress through
TranscriptionProgress::Heartbeat so the user sees "Retrying after 5s" rather than no signal.
(1) is the lowest-risk default for v1. (2) or (3) can land later when there's data on real-world transient-failure rates.
Where to make the change
crates/transcribe/src/openai_compat.rs::OpenAICompatibleTranscriber::new constructs the client via Client::with_config(config). The retry behavior is controlled at the reqwest layer; async-openai exposes Client::with_http_client(reqwest::Client) so a hand-built reqwest::Client (without ExponentialBackoff) can be substituted.
Definition of done
- Adapter constructed with retry disabled (or constrained)
- The 5xx wiremock test (currently omitted in
tests/openai_compat.rs) added back, exercising the BackendUnavailable mapping in < 5s
- Frontend UX unchanged for user-facing transient errors (still surfaces typed
TranscriptionError::BackendUnavailable)
Problem
async-openai 0.36's defaultClient<OpenAIConfig>ships withExponentialBackoffretry on all server errors (5xx, 429). The default configuration retries up to 8 times with growing backoff, totaling roughly 15 minutes wall-clock before surfacing the error.For an interactive transcription use case ("user clicked Transcribe and is staring at the spinner"), 15 minutes of silent retry is the wrong default. The user has no signal that anything is wrong; they don't know whether to wait, retry, or switch providers.
This came up in T4 implementation: a wiremock end-to-end test for the 5xx →
BackendUnavailablemapping took ~80s before the test runner even got the chance to assert. We worked around it by replacing the 5xx test with a 200-malformed-body test (JSONDeserialize→Internal), but the production behavior remains as-is.Suggested approach
Either:
TranscriptionProgress::Heartbeatso the user sees "Retrying after 5s" rather than no signal.(1) is the lowest-risk default for v1. (2) or (3) can land later when there's data on real-world transient-failure rates.
Where to make the change
crates/transcribe/src/openai_compat.rs::OpenAICompatibleTranscriber::newconstructs the client viaClient::with_config(config). The retry behavior is controlled at the reqwest layer; async-openai exposesClient::with_http_client(reqwest::Client)so a hand-builtreqwest::Client(withoutExponentialBackoff) can be substituted.Definition of done
tests/openai_compat.rs) added back, exercising the BackendUnavailable mapping in < 5sTranscriptionError::BackendUnavailable)