fix(ops): #60 cap + observe queryOpSet fan-out size

[JustMaier]

Summary

• Adds bitdex_query_op_set_fanout_size histogram + bitdex_query_op_set_rejected_total counter + bitdex_query_op_set_applied_slots_total counter so we can size a real cap from prod data instead of guessing.
• Adds BITDEX_QUERY_OP_SET_MAX_FANOUT env var. Default usize::MAX — ships with no enforcement so the histogram populates from real traffic. Operator sets a finite value (manifest tweak + roll) once the distribution is in.
• Skips the op (with tracing::warn! + counter increment) when fan-out > cap. Cursor advances; better than blocking the WAL reader forever or OOM'ing.
• Drive-by: src/relay/auth.rs test helper &str → &'static str to unblock cargo test --lib (separate commit 8fa2cd4).

Why

apply_query_op_set is invoked from the WAL reader thread to apply nested ops to every slot matching a queryOpSet's filter. The matching path:

let query = BitdexQuery {
    filters,
    sort: None,
    limit: usize::MAX,           // unbounded
    skip_cache: true,            // cold every call
    ...
};
let result = engine.execute_query(&query)?;
let slot_ids = &result.ids;
for &slot_id in slot_ids { ... per-slot ops ... }

No cap on result.ids length. Under burst load + prod's _RJEM_MALLOC_CONF=dirty_decay_ms:0,muzzy_decay_ms:0, each cold execute_query allocation is held in resident memory, ratcheting RSS. 2026-04-29 measurement: identical 50-fan-out burst on local rig produced +820 MB sustained, no decay with prod MALLOC_CONF vs -60 MB decay under default jemalloc settings. That's the smoking gun behind the v1.0.177-jemalloc shadow-on OOM (RSS 11.7 → 110 GB in <30s on talos-wjh-tgy).

#61 (jemalloc decay tuning) closes the allocator-side half of the loop. This PR closes the per-call workload half.

What's new

Metric	Type	Labels
bitdex_query_op_set_fanout_size	Histogram	index
bitdex_query_op_set_rejected_total	Counter	index, reason
bitdex_query_op_set_applied_slots_total	Counter	index
BITDEX_QUERY_OP_SET_MAX_FANOUT	Env var	default usize::MAX

Histogram buckets: [1, 10, 100, 1K, 10K, 100K, 1M, 10M, 100M].

Skip path emits:

WARN ops_processor: queryOpSet '<query>' matches <N> slots, exceeds cap <C> — op skipped (data drift)

Trade: skip semantics

When cap is exceeded the op is skipped silently (log + metric, no error to /ops POSTer). Cursor still advances. Drift consequence: the trigger that produced this queryOpSet (e.g. Post.publishedAt update fanning out to N Images) misses some rows. Acceptable because:

• Worst case: pop publishedAt update misses a few popular MVs briefly — re-runnable via reload.
• Better than blocking the WAL reader forever (cursor stalled = ALL ops blocked).
• Better than crashing the pod under memory pressure.

#62 (paginate-instead-of-skip) is the proper fix once we have a histogram-driven cap value.

Alert recommendation

rate(bitdex_query_op_set_rejected_total{reason="fanout_too_wide"}[5m]) > 0

Page when rejection rate > 0. Drift is invisible without this.

Test plan

• cargo check --features server,pg-sync --bin bitdex-server passes.
• test_max_fanout_default_is_usize_max — unset env yields usize::MAX.
• test_max_fanout_env_override_parses — BITDEX_QUERY_OP_SET_MAX_FANOUT=100000 yields 100K.
• test_max_fanout_invalid_env_falls_back_to_default — non-numeric env falls back to default.
• End-to-end cap-exceed path through real ConcurrentEngine — deferred to docs: generational cache invalidation design #62 since it requires the same fixture work paginate-instead-of-skip will need anyway.
• Histogram observation verified live once perf: generational doc cache eviction #61 ships and prod broadcasts data.

Refs

• Investigation: 2026-04-29 shadow-on flip OOM diagnosis on talos-wjh-tgy (v1.0.177-jemalloc → 110 Gi pod kill at T+30s).
• Companion PR: perf: generational doc cache eviction #61 jemalloc decay tuning (Tom).
• Follow-up: docs: generational cache invalidation design #62 paginate-instead-of-skip (proper fix; needs cap histogram data first).

🤖 Generated with Claude Code

[JustMaier]

Code review passed (Scarlet, via team channel). Histogram observed before cap check captures rejected upper tail. Default usize::MAX ships unenforced — populates histogram first, operator tunes from real distribution after 24h soak. Skip-on-overflow with cursor-advance is correct (acknowledged data drift, tracked as #62 paginate-instead-of-skip). 174 net lines, 5 files. 3/3 unit tests pass, cargo check passes. Ready to merge after scope decision (#61 manifest land + tonight relay-mode flip vs full server-flip).