Sd merge review fixes

.gitignore

@@ -5,3 +5,5 @@
 /.remember/
 /.claude/
 /coverage
+/benches/results.jsonl
+/benches/run_full.log

README.md

@@ -107,6 +107,11 @@ Order-of-magnitude numbers from local benchmarking against the legacy PHP engine
 These are single-machine, order-of-magnitude measurements, not precise benchmarks — treat
 them as a shape-of-the-win indicator, not a guarantee.
 
+For a reproducible, self-contained benchmark suite — rebuild / churn / search across
+1k–500k-document indexes, comparing the PHP extension against Zend Search Lucene on an
+identical deterministic corpus, with per-operation Rust heap and process RSS — see
+[`benches/`](benches/README.md) (`./benches/run.sh`).
+
 ## Continuous integration
 
 The pipeline in `.github/workflows/ci.yml` runs on every push/PR:

benches/README.md

@@ -0,0 +1,138 @@
+# sdsearch benchmarks
+
+A reproducible benchmark suite that measures the sdsearch engine on three workloads across
+index sizes of **1k / 10k / 50k / 100k / 500k** documents, and compares **our PHP extension
+against Zend Search Lucene** on the same corpus and the same queries.
+
+```bash
+# full run (needs the Zend oracle for the comparison — see Prerequisites)
+ZEND_LUCENE_PATH=/path/to/zend-framework-1/library ./benches/run.sh
+
+# fast smoke run (1k + 10k only)
+./benches/run.sh --quick
+```
+
+Output: `benches/results.jsonl` (one JSON line per measurement) and `benches/RESULTS.md`
+(rendered tables with cross-engine speedups). `results.jsonl` is machine-specific and
+git-ignored; regenerate it locally.
+
+## What is measured
+
+Three workloads, each as a function of index size N:
+
+| Workload  | What it does |
+|-----------|--------------|
+| `rebuild` | Build a **production-shaped** index from scratch: add N docs on top of the committed KB base, then `optimize()` to a single segment (as the host does per batch). |
+| `churn`   | On an optimized N-doc index, delete the first 1% of docs (by global id) and add 1% fresh docs, then commit (incremental update). |
+| `search`  | Query three term classes at two realistic paging depths (top-20, top-100) over the **optimized** index and report p50/p95 latency; the true hit count per class is measured separately. |
+
+Every index the benchmark searches or churns is `optimize()`d to a single segment first — the shape
+a production deployment uses — so query cost reflects the real deployment, not a many-segment
+work-in-progress index.
+
+Three engines:
+
+- **`native`** — the Rust core directly (`sdsearch-core/examples/bench_engine.rs`). This is the
+  only engine that can report the **exact Rust heap** high-water mark of an operation.
+- **`sdsearch`** — our PHP extension (`SdSearch\Writer` / `SdSearch\Engine`), i.e. exactly what a
+  PHP application uses.
+- **`zend`** — `Zend_Search_Lucene`, the pure-PHP engine sdsearch replaces.
+
+## Memory: three different numbers, on purpose
+
+- `heap_peak_kb` (**native only**): peak heap bytes attributed to the measured op alone (a
+  tracking global allocator, reset to steady-state right before the op). PHP's
+  `memory_get_peak_usage()` **cannot** see this, because the Rust heap lives outside the Zend
+  memory manager — that is why the native bench exists.
+- `rss_peak_kb` / `rss_peak_mb`: process peak resident memory (`VmHWM` from
+  `/proc/self/status`). It includes both engines' real footprint (the Rust heap shows up here),
+  so **RSS is the number to compare across engines**.
+- `php_peak_mb` (PHP engines): `memory_get_peak_usage(true)`. Meaningful for `zend`, but it
+  **understates `sdsearch`** (Rust heap invisible), so it is kept in `results.jsonl` for
+  reference and left out of the report tables.
+
+Each measurement runs in a **fresh process** so peak-RSS is not cross-contaminated. For `churn`
+and `search`, the PHP engines run against a **pre-built index** (an unmeasured `build` step), so
+the build's memory does not leak into the churn/search RSS. The native bench builds in-process
+but isolates the op's heap with the allocator reset, so its `heap_peak_kb` stays clean (its
+`rss_peak_kb` for churn/search may still include the build and is context-only).
+
+**Native runs in two passes.** The tracking allocator's atomic bookkeeping on every alloc/free
+taxes the hot path, so a heap-tracked run's wall time is not comparable to the extension's (which
+uses the plain system allocator). The runner therefore measures each native `rebuild`/`churn`
+twice: a **time pass** (`BENCH_TRACK_HEAP=0`, no atomics → honest `ms`, the engine floor without
+the FFI/JSON boundary) and a **heap pass** (`BENCH_TRACK_HEAP=1` → accurate `heap`, `ms` ignored).
+The report takes `ms`/`rss` from the time pass and `heap` from the heap pass. Absolute times are
+also sensitive to machine load (a busy laptop / thermal throttling adds noise, especially at
+small N where a run is only milliseconds); the **relative** ordering across engines is the stable
+signal — run on an idle, AC-powered machine for clean absolute numbers.
+
+## Corpus and queries (deterministic)
+
+The corpus is fully deterministic — no RNG — so runs are reproducible and the two engines index
+byte-identical documents. Doc `i` has a `title` (~5 tokens), a `body` (~40 tokens drawn from a
+fixed 25-word pool), and an `id` keyword. The generator is defined **identically** in
+`sdsearch-core/examples/bench_engine.rs` (`gen_docs`) and `tools/bench_compare.php` (`gen_one`);
+keep them in sync or the engines stop comparing the same work.
+
+Three tokens are planted to fix the search result classes at exact, size-independent doc
+frequencies (distinct 3-char prefixes so the fuzzy query path, `prefix_len = 3`, can never
+cross-match one class to another):
+
+| Class  | Token         | Appears in | Query result |
+|--------|---------------|------------|--------------|
+| `many` | `widetoken`   | every doc  | ~N hits |
+| `few`  | `sparsetoken` | 5 docs     | 5 hits |
+| `none` | `absenttoken` | never      | 0 hits |
+
+Searches use the same **fuzzy + wildcard + parser** boolean on all three engines (mirroring the
+engine's `text_subquery` and the host app's query builder), so hit counts match — the report's
+`hits` column doubles as a correctness gate: it must be identical across engines. Each query
+**reopens the index**, exactly as `SdSearch\Engine::search` and a fresh `Zend_Search_Lucene::open`
+do per request; at larger N the index-open cost dominates the low-hit classes. The native column
+uses the same `search_index` path, so it measures the engine's real per-request cost minus the
+FFI/JSON boundary. Zend's `find()` has no top-K limit (it always scores and returns everything),
+so its top-20 and top-100 latencies coincide.
+
+## Prerequisites
+
+- A Rust toolchain (builds `examples/bench_engine` and the PHP extension).
+- PHP 8.x CLI with the `sdsearch` extension available. If it is not already enabled in `php.ini`,
+  the runner loads `target/release/libsdsearch.so` with `-d extension=…` automatically.
+- **`ZEND_LUCENE_PATH`** pointing at a Zend Search Lucene `library/` root, for the `zend`
+  comparison. Without it, `zend` runs are recorded as `skipped` and the suite still reports
+  `native` + `sdsearch`.
+
+`./benches/run.sh` builds everything in release by default (pass `--no-build` to skip).
+
+## Zend at large sizes
+
+Zend indexes in pure PHP and is orders of magnitude slower, so large rebuilds are guarded:
+
+- `zend` is **skipped for sizes above `BENCH_ZEND_MAX_DOCS` (default 100000)**. Set
+  `BENCH_ZEND_MAX_DOCS=500000` to attempt the 500k point.
+- Every `zend` run has a wall-clock budget `BENCH_ZEND_TIMEOUT` (default 900s); over-budget runs
+  are recorded as `skipped`, never fatal.
+- For sizes at/above `BENCH_ZEND_ULIMIT_FROM` (default 500000) an address-space cap
+  `BENCH_ZEND_MEM_MB` (MB, `0` = off) is applied; an over-cap run fails its malloc and is skipped.
+
+`native` and `sdsearch` always run the full 1k–500k range.
+
+## Flags
+
+```
+--quick                 sizes = 1000 10000 only
+--sizes "1000 50000"    explicit size list
+--workloads "rebuild"   subset of {rebuild churn search}
+--engines "native zend" subset of {native sdsearch zend}
+--iters N               sampled search iterations (default 50)
+--no-build              skip the release build
+--append                append to results.jsonl instead of truncating
+```
+
+## Files
+
+- `benches/run.sh` — orchestrator (matrix, guards, collection, report).
+- `tools/bench_compare.php` — one measurement (engine × workload × size) → one JSON line.
+- `sdsearch-core/examples/bench_engine.rs` — the native bench (exact heap).
+- `tools/bench_report.php` — renders `RESULTS.md` from `results.jsonl`.

benches/RESULTS.md

@@ -0,0 +1,80 @@
+# sdsearch benchmarks
+
+Generated by `benches/run.sh` → `tools/bench_report.php`. Reproducible: deterministic corpus, fixed iterations, warm-up discarded. See [README.md](README.md) for the corpus and query definitions.
+
+| | |
+|---|---|
+| Date | 2026-07-12 05:27 |
+| Commit | `6c3c2b7` |
+| Host | Linux 7.0.0-27-generic |
+| CPU | 13th Gen Intel(R) Core(TM) i7-1355U |
+| PHP | 8.4.23 |
+
+**Engines.** `native` = the Rust core (examples/bench_engine) — reports the EXACT Rust heap. `sdsearch` = our PHP extension. `zend` = Zend_Search_Lucene (pure PHP).
+
+**Memory columns.** `heap` (native only) = peak Rust heap attributed to the op (`memory_get_peak_usage()` cannot see this). `rss` = process peak RSS (VmHWM), the cross-engine-comparable footprint. `php_peak` (PHP engines) omitted from tables because it understates sdsearch (Rust heap is invisible to it); it is in results.jsonl for reference.
+
+## rebuild — Rebuild (build the whole index from the KB base)
+
+| docs | native ms | native heap MB | native rss MB | sdsearch ms | sdsearch rss MB | zend ms | zend rss MB | speedup vs zend | doc_count |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 1,000 | 20.07 | 3.2 | 8.8 | 20.09 | 47.5 | 1,509.67 | 45.4 | 75.1× | 1,020 |
+| 10,000 | 139.60 | 3.2 | 12.7 | 185.15 | 51.1 | 18,128.12 | 51.6 | 97.9× | 10,020 |
+| 50,000 | 714.32 | 11.8 | 34.9 | 932.03 | 74.2 | 92,048.93 | 68.6 | 98.8× | 50,020 |
+| 100,000 | 1,340.99 | 23.5 | 65.5 | 1,816.75 | 104.5 | 209,928.19 | 91.9 | 115.6× | 100,020 |
+| 500,000 | 7,273.32 | 98.6 | 304.7 | 9,840.58 | 342.6 | 1,063,693.86 | 208.9 | 108.1× | 500,020 |
+
+## churn — Churn (delete 1% of docs + add 1% + commit)
+
+| docs | native ms | native heap MB | native rss MB | sdsearch ms | sdsearch rss MB | zend ms | zend rss MB | speedup vs zend | doc_count |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 1,000 | 0.25 | 0.2 | 8.6 | 1.06 | 43.5 | 6.66 | 44.5 | 6.3× | 1,020 |
+| 10,000 | 1.38 | 0.5 | 12.8 | 3.19 | 43.9 | 93.93 | 44.6 | 29.4× | 10,020 |
+| 50,000 | 6.54 | 1.7 | 35.1 | 11.34 | 45.3 | 461.22 | 44.8 | 40.7× | 50,020 |
+| 100,000 | 13.25 | 3.2 | 65.3 | 21.48 | 51.0 | 927.81 | 45.2 | 43.2× | 100,020 |
+| 500,000 | 73.97 | 9.3 | 305.7 | 111.76 | 65.5 | 5,861.84 | 46.2 | 52.5× | 500,020 |
+
+## search — term latency by result class (many / few / none)
+
+p50 latency in ms, measured at two realistic paging depths. `hits` is the TRUE match count for the class (must be identical across engines — correctness gate). Each query reopens the index, as the engines do per request; at larger N the index-open cost dominates low-hit queries. Zend's `find()` has no top-K limit, so its top-20 and top-100 latencies coincide.
+
+### top-20
+
+| docs | class | native p50 | sdsearch p50 | zend p50 | speedup vs zend | hits |
+|---:|---|---:|---:|---:|---:|---:|
+| 1,000 | many | 2.4733 | 1.1220 | 45.2299 | 40.3× | 1,000 |
+| 1,000 | few | 0.5714 | 0.5610 | 23.1121 | 41.2× | 5 |
+| 1,000 | none | 0.6404 | 0.6249 | 23.0150 | 36.8× | 0 |
+| 10,000 | many | 5.9287 | 5.5599 | 237.5951 | 42.7× | 10,000 |
+| 10,000 | few | 0.6820 | 0.6981 | 28.3499 | 40.6× | 5 |
+| 10,000 | none | 0.7604 | 0.7601 | 28.3911 | 37.4× | 0 |
+| 50,000 | many | 28.2357 | 26.2308 | 1,085.1121 | 41.4× | 50,000 |
+| 50,000 | few | 0.6883 | 0.6790 | 19.2802 | 28.4× | 5 |
+| 50,000 | none | 0.7198 | 0.6940 | 19.0599 | 27.5× | 0 |
+| 100,000 | many | 54.3672 | 59.6919 | 2,063.2579 | 34.6× | 100,000 |
+| 100,000 | few | 0.8188 | 0.8190 | 17.0100 | 20.8× | 5 |
+| 100,000 | none | 0.7656 | 0.7391 | 13.1478 | 17.8× | 0 |
+| 500,000 | many | 539.0156 | 559.2749 | 10,325.0971 | 18.5× | 500,000 |
+| 500,000 | few | 2.4895 | 2.3952 | 20.0241 | 8.4× | 5 |
+| 500,000 | none | 2.4090 | 2.2871 | 13.8309 | 6.0× | 0 |
+
+### top-100
+
+| docs | class | native p50 | sdsearch p50 | zend p50 | speedup vs zend | hits |
+|---:|---|---:|---:|---:|---:|---:|
+| 1,000 | many | 1.1545 | 1.2589 | 44.9741 | 35.7× | 1,000 |
+| 1,000 | few | 0.5645 | 0.5529 | 23.1988 | 42.0× | 5 |
+| 1,000 | none | 0.6391 | 0.6192 | 23.1290 | 37.4× | 0 |
+| 10,000 | many | 5.9922 | 5.6951 | 235.9731 | 41.4× | 10,000 |
+| 10,000 | few | 0.6676 | 0.7029 | 28.5289 | 40.6× | 5 |
+| 10,000 | none | 0.7790 | 0.7539 | 28.1141 | 37.3× | 0 |
+| 50,000 | many | 28.4841 | 26.7911 | 1,089.5112 | 40.7× | 50,000 |
+| 50,000 | few | 0.6864 | 0.6800 | 19.3150 | 28.4× | 5 |
+| 50,000 | none | 0.7140 | 0.7010 | 19.1391 | 27.3× | 0 |
+| 100,000 | many | 57.8445 | 58.9969 | 2,057.3859 | 34.9× | 100,000 |
+| 100,000 | few | 0.8201 | 0.7961 | 17.0660 | 21.4× | 5 |
+| 100,000 | none | 0.7655 | 0.7439 | 13.1581 | 17.7× | 0 |
+| 500,000 | many | 543.4759 | 561.6472 | 10,282.6638 | 18.3× | 500,000 |
+| 500,000 | few | 2.5341 | 2.3811 | 19.9978 | 8.4× | 5 |
+| 500,000 | none | 2.4181 | 2.3220 | 13.7968 | 5.9× | 0 |
+