[WIP] Default thin-client to enabled and add proxy connectivity-probe gate

[jeet1995]

Motivation

Gateway V2 (a.k.a. ThinClient) is the new Compute-hosted data plane for Cosmos DB. To roll it out without forcing every SDK consumer to opt in, we need a user-transparent way for the Java SDK to:

1. Default to ThinClient for eligible data-plane and QueryPlan requests so adoption tracks the federation roll-out automatically, and
2. Fall back to Gateway V1 the moment the ThinClient fleet looks unhealthy from the client's perspective — without surfacing the failure as a user-visible error and without per-request overhead.

Today the SDK has no client-side signal about ThinClient reachability. If a thin-client endpoint is degraded — bad HTTP/2 negotiation, TLS handshake failure, proxy 5xx — every eligible request is routed there and fails before falling back. This PR closes that gap by:

• Flipping COSMOS.THINCLIENT_ENABLED to true by default, so any account whose topology advertises thin-client read locations starts routing eligible traffic through Gateway V2 automatically.
• Adding an out-of-band connectivity probe (EndpointProbeClient) that POSTs to /connectivity-probe over the thin-client HTTP/2 transport on every topology refresh. The probe result is AND-ed into the existing routing gate (useThinClientStoreModel) via GlobalEndpointManager.isProxyProbeHealthy(). If the probe trips, eligible requests transparently route to Gateway V1 until probes recover — the caller never sees the difference except in CosmosDiagnostics.

Key invariants the design preserves:

• No per-request probe I/O. Probing is piggybacked on the existing topology-refresh cadence; the data-plane gate is a single in-memory boolean read.
• Optimistic startup. proxyHealthy = true until proven otherwise, so a slow first probe never delays the first request.
• Never trip client init. Every probe-wiring and probe-execution failure path is swallowed and logged — CosmosClient construction and topology refresh are protected by onErrorResume.
• Kill switch. COSMOS.THINCLIENT_PROBE_ENABLED=false short-circuits the probe HTTP I/O entirely while leaving routing on its last-known state (defaults to GREEN).
• Hysteresis. THINCLIENT_PROBE_FAILURE_THRESHOLD consecutive RED cycles to flip GREEN→RED; THINCLIENT_PROBE_RECOVERY_THRESHOLD consecutive GREEN cycles to flip back — prevents routing oscillation.

Call sequence

sequenceDiagram
    autonumber
    participant App as User App
    participant Client as CosmosClient<br/>(RxDocumentClientImpl)
    participant GEM as GlobalEndpointManager
    participant Probe as EndpointProbeClient
    participant H2 as ThinClient HTTP/2<br/>HttpClient
    participant Proxy as ThinClient<br/>Endpoint (Gateway V2)
    participant V1 as Gateway V1<br/>(fallback)

    rect rgba(200,220,255,0.25)
    Note over App,Probe: Bootstrap (one-time)
    App->>Client: new CosmosClient(...)
    Client->>Client: useThinClient = config.isThinClientEnabled()  (now defaults true)
    Client->>GEM: setThinClientHttpClient(reactorHttpClient)
    GEM->>Probe: new EndpointProbeClient(httpClient)
    Note over Probe: proxyHealthy = true (optimistic)
    end

    rect rgba(200,255,210,0.25)
    Note over GEM,Proxy: Topology refresh (every refresh tick)
    GEM->>GEM: refreshLocationPrivateAsync()
    GEM->>GEM: runThinClientProbeCycleMono()
    GEM->>Probe: runProbeCycle(thinClientRegionalEndpoints)
    par Per regional endpoint
        Probe->>H2: POST <endpoint>/connectivity-probe
        H2->>Proxy: HTTP/2 frame
        Proxy-->>H2: 200 OK (or error / timeout)
        H2-->>Probe: ProbeResult(GREEN|RED)
    end
    Probe->>Probe: applyCycleResult() — apply hysteresis<br/>update proxyHealthy
    Note right of Probe: ALL endpoints 200 → GREEN<br/>any non-200 / timeout → RED<br/>flip only after N consecutive
    end

    rect rgba(255,235,200,0.25)
    Note over App,V1: Data-plane / QueryPlan request
    App->>Client: documents().readItem(...) / queryItems(...)
    Client->>Client: useThinClientStoreModel(request)
    Client->>GEM: isProxyProbeHealthy()
    GEM->>Probe: isProxyHealthy()
    Probe-->>GEM: true (GREEN) / false (RED)
    GEM-->>Client: gate result
    alt useThinClient && hasThinClientReadLocations && proxyHealthy && eligible
        Client->>Proxy: route through ThinClient (Gateway V2)
        Proxy-->>Client: response
    else gate failed
        Client->>V1: route through Gateway V1
        V1-->>Client: response
    end
    Client-->>App: response (transparent fallback)
    end

Loading

What's changing

Area	Change
Default	COSMOS.THINCLIENT_ENABLED defaults to true
New class	com.azure.cosmos.implementation.EndpointProbeClient — probe lifecycle, hysteresis, single-flight via cycleInProgress CAS, never-throw contract, body-drain inside the Mono lifecycle so reactor-netty releases pooled connections
Wiring	GlobalEndpointManager.setThinClientHttpClient(...) invoked from RxDocumentClientImpl when useThinClient is true; runThinClientProbeCycleMono() chained into refreshLocationPrivateAsync
Routing gate	RxDocumentClientImpl.shouldUseThinClientStoreModel(...) now ANDs in isProxyProbeHealthy(); stored procedures (isExecuteStoredProcedureBasedRequest()) explicitly eligible
Diagnostics	EndpointProbeClient.DiagnosticsSnapshot exposes last cycle result, consecutive-failure/success counters, last-state-change timestamp; reachable via GlobalEndpointManager.getThinClientProbeDiagnostics()
Safeguard	If hasThinClientReadLocations=true but the resolved endpoint set is empty (eligibility/resolution mismatch), forceUnhealthy(...) pins the gate to RED so traffic falls back to Gateway V1

Configuration

Property	Env var	Default	Purpose
COSMOS.THINCLIENT_ENABLED	COSMOS_THINCLIENT_ENABLED	true (changed)	Master switch for thin-client routing
COSMOS.THINCLIENT_PROBE_ENABLED	COSMOS_THINCLIENT_PROBE_ENABLED	true	Kill switch for probe HTTP I/O. When false, no probes fire; routing stays at last-known state (defaults to GREEN/optimistic)
COSMOS.THINCLIENT_PROBE_FAILURE_THRESHOLD	COSMOS_THINCLIENT_PROBE_FAILURE_THRESHOLD	1	Consecutive RED cycles before GREEN→RED
COSMOS.THINCLIENT_PROBE_RECOVERY_THRESHOLD	COSMOS_THINCLIENT_PROBE_RECOVERY_THRESHOLD	1	Consecutive GREEN cycles before RED→GREEN
COSMOS.THINCLIENT_PROBE_PATH	COSMOS_THINCLIENT_PROBE_PATH	/connectivity-probe	Probe URI path
COSMOS.THINCLIENT_CONNECTION_TIMEOUT_IN_MS	COSMOS_THINCLIENT_CONNECTION_TIMEOUT_IN_MS	(existing)	Per-probe deadline

Testing

• Unit tests: EndpointProbeClientTests, ThinClientProbeWiringTests
• E2E: ThinClientStoredProcedureE2ETest, ThinClientQueryE2ETest, Http2PingKeepaliveTest (probe disabled in pure-Gateway-V1 branch only — see review item 4)
• Live validation against endpoint-probe-49437 (NEU, single-region thin-client-enabled account): 20-minute Mixed 90/9/1 benchmark with -DCOSMOS.HTTP2_ENABLED=true -DCOSMOS.THINCLIENT_ENABLED=true confirms probes fire (Http2PingHandler installed + ThinClient log emissions verified in process logs); 196/199 thinclient profile tests pass.

Independent benchmark validation (probe-off c-sweep, 2026-06-13)

Second-source perf comparison vs main on a different account (thin-client-mr-eventual-ci, WCUS, multi-region Eventual — distinct from the endpoint-probe-49437 NEU account in the existing Testing section) and a different VM (Standard_D4s_v5, westcentralus, colocated). Probe gate disabled (rationale below).

Setup

• VM: Standard_D4s_v5 (4 vCPU, 16 GiB), Ubuntu 22.04, westcentralus
• Account: thin-client-mr-eventual-ci
• SDK flags: -DCOSMOS.THINCLIENT_ENABLED=true -DCOSMOS.HTTP2_ENABLED=true -DCOSMOS.THINCLIENT_PROBE_ENABLED=false
• Refs: PR head be37e742c36 vs main HEAD 175199ec727
• Workload: BenchmarkConfig JSON, connectionMode=GATEWAY, consistencyLevel=Eventual, maxRunningTimeDuration=PT1H, ReadThroughput
• Routed via :10250 (ThinClientStoreModel) on every run — confirmed in benchmark.log

Results

Concurrency	main	PR #49437	Δ
4	2,439 ops/s · 1.59 ms	2,160 ops/s · 1.80 ms	PR −11.4 % (single-run variance — see below)
20	9,882 ops/s · 1.98 ms	9,878 ops/s · 1.98 ms	tied (+0.04 %)
50	13,358 ops/s · 3.46 ms	13,714 ops/s · 3.46 ms	PR +2.7 %

Health (every run)

• 0 FATAL / 0 ERROR (workload phase) / 0 caught exceptions / 0 non-2xx
• Heap flat (4.9 GB peak / 8 GB max), no thread/FD leak, GC < 0.04 % CPU
• Toolkit metrics-check.json allPassed=true for all 6 runs

Conclusion

No actionable perf regression at the meaningful concurrency points (c=20, c=50). The c=4 −11.4 % is single-run variance — it did not reproduce at c=20 (tied) and went the other way at c=50 (PR +2.7 %).

Caveat — probe disabled (COSMOS.THINCLIENT_PROBE_ENABLED=false)

The server-side enableConnectivityProbe federation flag is currently OFF on every PROD *-fe federation our test accounts live on. Verified via the CosmosDB repo (code default false in FederationConfiguration.ThinProxy.cs:446; no Phase-2 enablement PR has merged on either releases/EN20260506 or releases/EN20260409). The proxy returns 503 to POST /connectivity-probe, the SDK probe gate (correctly) marks UNHEALTHY on cycle 1, and data plane falls back to Gateway V1 — meaning we would measure fallback throughput, not thin-client throughput.

To exercise the actual :10250 data path I disabled the SDK-side probe. The probe gate behavior itself should be re-validated on a federation where the server flag is true (the test4 fleet that server PR 2107592 validated against) before this client PR ships to customers — otherwise customers will see Gateway V1 fallback in every region where Phase 2 has not rolled out yet.

Logs

Result directories preserved on the benchmark VM at /home/azureuser/azure-sdk-for-java/sdk/cosmos/azure-cosmos-benchmark/results/20260613-SIMPLE-{main,PR-49437}-{commit}-c{4,20,50}/. Each contains benchmark.log, gc.log, monitor.csv, workload-config.json, metrics-check.json, and per-metric Micrometer CSVs.

[jeet1995]

Build 6424287 went from 49+/2 distinct test failures down to two new test-only failure patterns, both rooted in the same cause — the probe gate (enabled by default, threshold=1) is flipping data-plane routing from the thin-client proxy to Gateway V1 in test accounts whose proxy has not yet deployed the /connectivity-probe endpoint.

New failures (both fixed in ad9e3df):

1. CosmosNotFoundTests.performBulkOnDeletedContainerWithGatewayV2 (45 failures, log 1986) — asserts response substatus 1003 from the thin-client routing path, observed 0 because requests went to Gateway V1.
2. PerPartitionCircuitBreakerE2ETests.*Gateway (26 failures, log 2002) — TestSuiteBase.assertThinClientEndpointUsed could not find any request whose endpoint contained :10250/.

Fix: Disable the probe by default in TestSuiteBase's static initializer (only when the property is not already set), so all E2E tests inherit deterministic configuration-driven routing. Dedicated probe tests (EndpointProbeClientTests, ThinClientProbeWiringTests) set the property explicitly in @BeforeMethod and are unaffected. The per-class override in Http2PingKeepaliveTest is now redundant and was removed (its @AfterClass clear would have re-enabled the probe for any later E2E test sharing the JVM).

No production impact — these are test-environment changes. The probe still defaults to true in production via the DEFAULT_THINCLIENT_PROBE_ENABLED config; customers running with the property unset will get the probe enabled.

Remaining single-shot failures (likely environmental, not probe-related):

• CosmosTracerTest.cosmosAsyncDatabase ThreadTimeoutException at 40s (log 4742) — Direct TCP mode, probe doesn't gate Direct routing. Watching next run.
• DocumentQuerySpyWireContentTest.before_DocumentQuerySpyWireContentTest 429 RequestRateTooLarge (log 2251) — @BeforeClass setup throttling on shared test account; user-agent in the error correctly shows |F4 suffix confirming user-agent helper fix is working.
• OrderbyDocumentQueryTest.before_OrderbyDocumentQueryTest 404 "Collection is not yet available for read" (log 2696) — proxy-side propagation race on freshly created collection.

Commit: ad9e3df

[jeet1995]

Update on the CI-failure fix for build 6424287

My earlier comment proposed disabling the connectivity probe globally in TestSuiteBase. That was the wrong call --- the probe is ON by default in production (default failure threshold = 1), and tests should reflect production behaviour rather than mask it.

What I just pushed (commits e381f3a4a26 revert + da6c6983b80):

1. Reverted the global probe-disable in TestSuiteBase so production-equivalent defaults are restored for the broad test population.
2. Added a per-class probe-disable only to the three test classes whose assertions explicitly require the data plane to land on the proxy:
   • CosmosNotFoundTests (thinclient group) --- asserts OWNER_RESOURCE_NOT_EXISTS / assertThinClientEndpointUsed.
   • PerPartitionCircuitBreakerE2ETests (fi-thinclient-multi-master group) --- asserts assertThinClientEndpointUsed for gateway-mode traffic.
   • Http2PingKeepaliveTest --- installs iptables DROP on port 10250, which would also defeat the probe before the PING handler fires. (Disable restored by the revert.)

Each @BeforeClass sets COSMOS.THINCLIENT_PROBE_ENABLED=false before client construction and the corresponding @AfterClass clears it so the JVM's other test classes are not polluted. Every other test class continues to run with the production default (probe ON, threshold = 1).

This keeps the rest of the suite honest about default behaviour while preventing the three classes that need a deterministic proxy route from failing while the proxy-side /connectivity-probe endpoint is still rolling out across CI accounts.

[jeet1995]

Deep review summary

Item 1 — Stored Procedure routing on thin-client ✅

Supported. RxDocumentClientImpl.shouldUseThinClientStoreModel (lines 9023–9047) routes isExecuteStoredProcedureBasedRequest() via the early-return on line 9033 and the inclusion list on line 9045. New ThinClientStoredProcedureE2ETest covers it end-to-end.

Item 2 — Mono.<Void>empty() at GlobalEndpointManager.refreshLocationPrivateAsync (lines 322, 337)

Necessary, not stylistic. The Mono.defer(...) lambda has three return paths (cached Mono, deferred Mono, Mono.empty()). Without the type witness, Mono.empty() infers Mono<Object> because Mono is invariant in its type parameter — and the outer method returns Mono<Void>, so the lambda fails to type-check. Suggestion: keep the witness; a one-line comment ("type witness needed; Mono is invariant") would make this self-documenting.

Item 3 — azure-cosmos-tests/pom.xml orphan AAD line ✅ fixed

The thinclient profile referenced ${cosmos.use.aad.auth}, a Maven property defined nowhere in the repo (grep across all pom.xmls confirms). It would either no-op or interpolate to a literal. Removed in 56efc37.

Item 4 — Http2PingKeepaliveTest probe disable ✅ fixed

Probe disable now gated if (!THIN_CLIENT_ENABLED) in both @BeforeClass and @AfterClass. Rationale captured in the source comment: port 10250 = thin-client, port 443 = pure Compute Gateway.

• Pure CG branch (THIN_CLIENT_ENABLED=false): probe POSTs to 10250 are pure noise — they cannot influence routing and only add traffic/log clutter, so disabling is correct.
• Thin-client branch (THIN_CLIENT_ENABLED=true): probe stays enabled so the production probe gate is exercised by the test.

Applied in 56efc37.

Item 5 — COSMOS.THINCLIENT_PROBE_ENABLED kill-switch ✅ validated

Works as intended: when false, EndpointProbeClient.maybeRefreshAsync() short-circuits to Mono.empty() (lines 109–127) so zero probe HTTP I/O occurs. Routing then keeps using whatever proxyHealthy is set to (default true, line 71) — so the switch kills the probe wire calls without disabling thin-client routing. That is the correct kill-switch semantic.

FYI (non-blocking): forceUnhealthy() (line 177) flips proxyHealthy = false with no recovery counterpart. Today it is only invoked from the probe path itself, so disabling the probe makes that path unreachable — safe by construction. Worth a one-line comment near forceUnhealthy documenting that invariant, in case anyone ever calls it from a different code path with the probe disabled.

---

Items 3 and 4 are pushed as 56efc37 on this branch.

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[jeet1995]

/azp run java - cosmos - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).