perf(filter): chunk bitmap_bytes iteration to bound read-lock-hold

[JustMaier]

Summary

FilterField::bitmap_bytes() is called by the bitmap memory scanner thread every time a field is flagged stale. The previous single-lock impl held the bitmaps read lock for the entire iteration: ~50 ms on userId (769K entries), ~1–2 s on postId (22.5M).

Pre-#224 the merge thread's mark_stale calls were unreachable (the stray } bug), so the scanner had nothing to re-scan and bitmap_bytes was effectively dormant. Post-#224 the merge thread runs the full cycle and the scanner re-reads at full cadence — turning bitmap_bytes into the dominant write-lock blocker for the apply path.

Root cause (instrumented bisect)

Apply path's remove_bulk was logging lock=80-235ms mut=4-15μs. The work was microseconds; apply was waiting on a reader.

Added 5 ms threshold logs on every read-lock acquire + hold across apply_diff_eq, union_with_diff, for_each_versioned, iter_versioned, merge_dirty, and remove_bulk itself. Only bitmap_bytes's iteration showed the long hold; everything else was under 5 ms.

Fix

Walk the field in 16K-key chunks, releasing the read lock between chunks. Same chunked-iter pattern that PR #225 uses for save_snapshot and range_scan. Single-line semantic change to the iteration shape.

Wall-clock is unchanged (same per-VB work in aggregate). The win is bounded max-continuous-lock-hold for queued writers.

Measurements (local, sustained PG-sync stream, 100 s)

Metric	Pre-fix	Post-fix
[remove_bulk_slow] lock_us distribution	8–235 ms (mean 76 ms)	5–11 ms (mean 8.6 ms)
Apply mean (small batches)	146 ms	< 10 ms
Apply mean (WAL catch-up batches: 30K–150K ops)	146 ms	110–160 ms (legitimate insert work, not contention)
[read_slow] for_each_versioned_chunked.chunk hold_us (new visible)	n/a	~10 ms per 16K-entry chunk

Stacking

• Branched off main directly. No dependency on other open PRs.
• Pairs with fix(merge_thread): close while-loop brace correctly so post-cursor work runs every cycle #224: this fix is what unblocks fix(merge_thread): close while-loop brace correctly so post-cursor work runs every cycle #224's merge. Together they give us PR perf(filter): track dirty values in side set so merge_dirty avoids O(N) walk #222's apply-mean win back without the read-lock-blocker that fix(merge_thread): close while-loop brace correctly so post-cursor work runs every cycle #224's reanimated mark_stale introduced.
• Independent of fix(ops_wal): skip unreadable tail on closed gen to break reader stall #221, perf(filter): track dirty values in side set so merge_dirty avoids O(N) walk #222, perf(snapshot): parallelize per-bucket writes in save_snapshot #223, perf(snapshot): chunk for_each_versioned to bound lock-hold under high cardinality #225, perf(prefilter): skip 0-cardinality auto-promoted prefilters #226 (all already merged to main per Scarlet's batch).

Test plan

• Build clean (cargo build --profile fast --features server,pg-sync)
• Live measurement with PG-sync streaming + bitmap memory scanner active — apply lock_us dropped 10-20x
• Stacked re-measurement after merge (pending)
• Time-to-first-warm-restore (pending — separate task)

Future work (deferred)

Cache the byte count with a TTL or update incrementally on mutation, eliminating the periodic re-scan entirely. Filed as a follow-up.

🤖 Generated with Claude Code

[JustMaier]

Reviewed. Approved.

Diff matches diagnosis: snapshot keys upfront under brief read-lock, then chunked iteration with per-16K-chunk read-lock acquire/release. Apply's writers get to interleave between chunks instead of blocking the full scan window.

Concerns / things to watch (none blocking):

1. Key-snapshot allocation cost — r.keys().copied().collect() allocates O(N × 8) bytes upfront. For postId at 22.5M entries that's ~180 MiB per scan call. Bounded (memory scanner runs every few seconds, not hot path), and trade is memory spike for lock-hold reduction. Acceptable. Your TODO to cache bitmap_bytes with TTL or update incrementally is the right longer-term fix.

2. Approximate vs exact — between key-snapshot and chunk iteration, keys can be added (not counted) or removed (r.get returns None — properly handled). bitmap_bytes is now approximate under concurrent mutation, but this is a metrics scrape, exact isn't required.

3. Why this differs from the abandoned per-key-lock variant — old comment captured that pattern. Donovan's chunked impl correctly differentiates (16K chunks vs per-key) so it doesn't recreate the writer-starvation issue. Comment update is clear about that distinction.

4. Per-chunk lock-hold ~10 ms × ~1400 chunks (postId 22.5M / 16K) = ~14 s wall-clock per scan. Long total, but acceptable since each chunk releases. Apply writers no longer queue behind a single 1-2 s reader.

Approved. Same self-approve caveat — treat this comment as approval. Merging next + tagging v1.0.169-jemalloc, then merging #224 paired.