perf(events): h2c multiplexing client for internal event hops

[pingsutw]

Draft — opening early for visibility; benchmarking in progress on dev.

Why are the changes needed?

The executor→events-proxy and events-proxy→run-service hops are built with http.DefaultClient:

// executor/setup.go
eventsClient := workflowconnect.NewEventsProxyServiceClient(http.DefaultClient, ...)
// events/setup.go
runClient := workflowconnect.NewInternalRunServiceClient(http.DefaultClient, ...)

http.DefaultClient uses HTTP/1.1 with MaxIdleConnsPerHost = 2. Under concurrent reconciles it doesn't cap concurrency (so throughput isn't hard-limited) but it churns short-lived connections — repeatedly opening/closing TCP connections, which drives connection-setup overhead, ephemeral-port/TIME_WAIT pressure, and p99 tail latency on the event-send path under sustained load.

The internal servers already advertise unencrypted HTTP/2 (h2c) via httpProtocols() (SetUnencryptedHTTP2(true)), and these hops are plaintext http://, so we can use h2c on the client side for free and let all concurrent RPCs multiplex over a stable connection.

What changes were proposed in this pull request?

• Add app.InternalHTTPClient() (next to the server's httpProtocols()): an *http.Client using native h2c via http.Transport.Protocols (Go 1.24+), with a generous idle pool as h1 fallback.
• Use it on both internal event hops (executor/setup.go, events/setup.go) in place of http.DefaultClient.

No API or behavior change — purely the transport.

How was this patch tested?

go build / go vet clean. Isolated micro-benchmark of the two clients against an h2c server (equal modeled dial cost + server delay, concurrency 200):

DefaultClient  throughput=23200 req/s  p50=8.1ms  p99=19.2ms  conns_dialed=2443
h2c-tuned      throughput=22571 req/s  p50=8.6ms  p99=14.4ms  conns_dialed=200

Headline is 12× fewer connection dials (2443 → 200) at equal throughput and a tighter p99. On localhost the latency delta is modest because handshakes are nearly free; the real-cluster benefit (TIME_WAIT / handshake-RTT under sustained cross-pod load) is being measured on dev and will be added here.

Note: this is connection-churn/p99 hygiene — it is not the primary throughput lever for the event pipeline. The bigger wins are batching events per RPC and a single multi-row INSERT (separate follow-up).

Labels

• changed

Check all the applicable boxes

• All new and existing tests passed.
• All commits are signed-off.

[Copilot]

Pull request overview

This PR introduces a dedicated internal service-to-service *http.Client to reduce connection churn by enabling cleartext HTTP/2 (h2c) multiplexing for Connect RPC calls on the executor → events-proxy and events-proxy → run-service hops.

Changes:

• Added app.InternalHTTPClient() with an h2c-enabled transport and larger idle connection pool.
• Updated executor and events setup to use app.InternalHTTPClient() instead of http.DefaultClient for internal Connect clients.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
flytestdlib/app/app.go	Adds InternalHTTPClient() used for internal Connect RPC traffic (h2c + tuned transport).
executor/setup.go	Switches EventsProxy Connect client from http.DefaultClient to app.InternalHTTPClient().
events/setup.go	Switches InternalRunService Connect client from http.DefaultClient to app.InternalHTTPClient().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.