landing-variant A/B test not splitting — all visitors get free-plus fallback

[kilbot]

Summary

The landing-variant A/B test is not splitting traffic — every real visitor renders the free-plus fallback. The PostHog flag is configured correctly and resolves a clean ~50/50 split server-side; the failure is in client-side variant resolution (src/bootstrap/variant-loader.ts). This bundle owns the experiment; the server-side enabler lives in the plugin.

Companion issue (plugin-side init/bootstrap fix): wcpos/woocommerce-pos#1188

Evidence (from production PostHog / ClickHouse on ph.wcpos.com)

landing_variant_rendered, all-time:

render_source	variant	count	who
fallback	free-plus	337	every real visitor
cache	indie	2	manual test hits only

• $feature_flag_called for landing-variant: 164 calls / 138 users, $feature_flag_response empty every time (client gets undefined).
• Zero render_source: "flag" events exist — the cold path never resolves a network value.
• Server-side get_all_feature_flags() returns a clean ~50/50 indie/free-plus split, so the flag itself works.
• Flag config: active: true, rollout 100%, variants indie 50 / free-plus 50, ensure_experience_continuity: True.

Root cause

prepareVariant / resolveFlag in src/bootstrap/variant-loader.ts decide the variant client-side after load:

• Cold path waits up to 500 ms (resolveFlag(ph, timeoutMs = 500)) for onFeatureFlags. With ensure_experience_continuity: True the flag requires a server round-trip that doesn't beat 500 ms against the self-hosted instance → flagValue: undefined → fallback to free-plus (render_source: fallback).
• Compounding it, the plugin (woocommerce-pos/includes/Admin/Menu.php) calls posthog.identify(distinctId) at init before this bundle loads — which violates this bundle's deliberate flag-before-identify ordering (spec §5.1) and poisons anon-bucket exposure for an experience-continuity flag.

So first-time visitors wait ~500 ms and still get the wrong variant; only cached repeat visitors render a real assignment.

Fix (this repo)

The decision should not block on a browser-side network call. Preferred direction is consume a server-injected bootstrap rather than racing the flag fetch:

1. When the plugin bootstraps landing-variant into posthog.init (see companion issue), getFeatureFlag(FLAG_KEY) returns synchronously and the cold path resolves with render_source: flag instantly — no 500 ms wait, no flicker. Verify variant-loader.ts handles a bootstrapped value cleanly through the flag path (and that exposure accounting still fires once, in the right bucket).
2. Re-evaluate the 500 ms cold-path timeout as a fallback only path. Raising it alone is not the fix (it trades a wrong variant for a slow one) — the goal is to not depend on the in-browser fetch for the decision at all.
3. Confirm flag-before-identify ordering holds once the plugin stops identifying early.

Also (no deploy)

Turning off ensure_experience_continuity on the flag in PostHog removes the forced server round-trip and may let the current cold path succeed — useful to confirm the diagnosis independent of any bundle change.

Acceptance

• New landing_variant_rendered events: ~50/50 split, render_source: flag.
• $feature_flag_called for landing-variant has a non-empty response.
• First-time visitors render the assigned variant with no perceptible wait / no flash of default.

[kilbot]

✅ Done (no-deploy step): ensure_experience_continuity has been turned off on the landing-variant flag in production PostHog (ph.wcpos.com) on 2026-06-19, and the team flag cache was refreshed. Server-side evaluation now returns a clean 20/20 indie/free-plus split over 40 sample anon visitors.

This removes the forced server round-trip that was making the client's 500 ms cold path time out. Watch landing_variant_rendered over the next day — fresh visitors should start producing render_source: flag events with a ~50/50 split instead of all fallback/free-plus. If that happens, it confirms the diagnosis; the bootstrap + identify-ordering work here is still the proper flicker-free/zero-wait fix and should still ship.

[kilbot]

⚠️ Correction — real root cause is server-side infra, not the client or flag config

Re-checked production a week after the ensure_experience_continuity change. It did not help — still 100% fallback/free-plus, $feature_flag_response still empty. Deeper investigation shows the earlier client-timing / experience-continuity diagnosis was wrong. The actual cause:

The PostHog feature-flags Rust service cannot serve any flags — its DB connection pool is deadlocked.

Evidence chain (production, ph.wcpos.com):

• /flags/?v=2 returns 401 "API key invalid" to every browser (and to curl). The token is valid.
• feature-flags logs: Token validation failed ... TimeoutError(Some("pool_timeout")), http_status: 401, error_code: "invalid_token", duration_ms ~3002 — 2,443 consecutive failures since boot, never one success.
• Postgres is healthy (31/100 connections) and the web/Django service uses it fine.
• From the feature-flags container's exact network namespace, a real libpq client connects + authenticates + runs select 1 over IPv4, IPv6, and by hostname — all succeed. So network, DNS, credentials, and Postgres are all fine.
• strace of the worker under live traffic: it does accept4/recvfrom/writev (serving the 401s) but makes zero connect() calls — it never attempts a Postgres connection. Every request blocks on the pool semaphore (futex) for the 2s acquire_timeout and returns pool_timeout.
• docker restart of the service did not fix it — a fresh process deadlocks again within seconds.

Impact: ALL PostHog feature flags are down site-wide (every flag, every user), and have been for the life of this service — not just this A/B test.

This means the code changes in this issue are not the fix and not the priority. The landing/plugin bootstrap work is still worthwhile (zero-wait, flicker-free), but it is moot until the flags backend serves values. The fix is infrastructure-side (PostHog feature-flags service pool config / image), tracked separately. Keeping this issue open for the client-side hardening, but de-prioritized behind the backend fix.