perf: reuse OpenAI client and add undici keep-alive Agent with connection warmup#100
Merged
qorzj merged 9 commits intoMay 21, 2026
Merged
Conversation
Cache the OpenAI client at module level keyed by (apiKey, baseURL) to avoid creating a fresh HTTP connection pool on every LLM turn. The client is a stateless fetch wrapper so sharing across calls is safe. Model, thinking-mode and other settings are still read fresh from config files each time. Also add a mount-time warmup effect that eagerly creates the client so the TCP+TLS connection is established while the user composes their first prompt.
The default undici-based global fetch only keeps connections alive for 4 seconds, which is too short for a CLI where the user may spend 10–30 seconds reading output before typing the next prompt. Add a custom fetch implementation backed by node:https.Agent with keepAlive: true and a 60-second idle timeout. The custom fetch is passed to the OpenAI SDK constructor so every LLM API request benefits from persistent connections across conversational turns. Also handles streaming request bodies (ReadableStream) for SDK features like file uploads.
Use npm undici's Agent with keepAliveTimeout: 60s instead of the 90-line custom https.Agent-based fetch wrapper. The approach is the same but much simpler — just pass undiciFetch with a configured Agent dispatcher to the OpenAI SDK.
Required by the custom fetch wrapper that replaces the default 4s keepAlive undici global dispatcher with a custom Agent (60s).
undici is imported at runtime in App.tsx for the custom keepAlive Agent. When bundled with --packages=external, end users need the package installed — it cannot be a devDependency.
undici v8 requires Node >=22, but the CI matrix includes Node 20 which the project intentionally supports. v7 works on >=20.18.1.
Codex review found that the fire-and-forget warmup models.list() had no timeout. The OpenAI client defaults to a 10-minute timeout, so an unreachable API could keep the Node process alive long after the user exits.
…tion warmup Extract OpenAI client creation logic into src/common/openai-client.ts: - Custom undici Agent with 60s keepAlive timeout (default is 4s) - Module-level client instance cache (reuse across calls) - Fire-and-forget connection warmup on first creation (3s timeout) - getMachineId() helper The App.tsx now simply imports and re-exports createOpenAIClient from the new common module, keeping UI concerns separate from HTTP/client lifecycle management.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
优化 OpenAI API 客户端的 HTTP 连接复用，减少用户输入间隙的 TLS 握手开销。
背景
性能收益
每次 LLM 调用（chat、tool use 等）减少 80ms~100ms，冷启动首字节延迟从约 210ms 降至约 130ms，降幅约 38%~43%。
可以参考 LOL、王者等游戏的 200+ms 延迟降低到 130ms，体感上这种延迟降低可以改善使用上的手感。
从绝对值来说，100 次 chat/tool_use 大概可以节约 10s，这对长任务来说也有一定的改善收益。
改动
keepAliveTimeout: 180s）替换默认 fetchsrc/common/openai-client.ts，保持 App.tsx 精简文件变更
package.jsonundici: ^7.25.0src/common/openai-client.tssrc/ui/App.tsxsrc/ui/index.ts检查