fix(eval,detect): correct bench gold, gate geolocation, close URL bypass

CHANGELOG.md

@@ -3,6 +3,137 @@
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] — bench correctness + detection fixes
+
+Started as gateway hardening from a Claude Code proxy integration
+report. Two of the four filters that report motivated did not survive
+measurement and were dropped; benchmarking them surfaced a defect in
+the benchmark itself that had been mis-ranking every tool in the
+published table. Public API surface unchanged.
+
+### Behavior changes
+
+- **Placeholder-escape PUA sentinels stripped before detection**
+  (#43). User-authored `{{...}}` templates wrapped in PUA sentinels
+  (`U+E000` / `U+E001`) used to be tagged as `private_person` by GLiNER
+  when `any-ascii` left the codepoints in the normalized text. Vault
+  substitution then corrupted the user's `{{...}}` into
+  `{{PII_PRIVATE_PERSON_*}}...{{`. Now stripped in
+  `normalizeForDetection` via the existing `ZERO_WIDTH_CHARS` set.
+- **`private_geolocation` gated on coordinate shape.** The schema
+  defines the label as a coordinate — all three recognizers that own it
+  match lat/lon literals, and `core:geo-latlon-decimal` range-validates.
+  The GLiNER head is prompted with the label zero-shot and never trained
+  on it, so it was applying it to place names, regions and postcodes:
+  only **7.6%** of its emissions on `ai4privacy-300k-heldout` were
+  coordinate-shaped, the rest `Deutschland`, `Sachsen`, `52396`. Every
+  gold span of the class is a lat/lon pair, so the gate removes false
+  positives without touching a conforming true positive — measured
+  **−2,317 FP, ±0 TP**, precision 0.129 → 0.902, recall unchanged. Runs
+  before cross-label dedupe, so the model's competing `private_address`
+  candidate survives the overlap instead of losing it to a label it
+  should never have won (+83 recovered TP: `private_ip` +60,
+  `private_address` +11, `private_vehicle_id` +7, `private_mac` +5).
+- **Public-URL allowlist on by default** (#45). URLs targeting
+  `PUBLIC_URL_HOSTS` (github / gitlab / docs.python.org / anthropic /
+  openai / huggingface / wikipedia / mdn / stackoverflow / cloud
+  vendor docs / standards bodies — full set in `src/url-filter.ts`)
+  are dropped from `private_url` output. Subdomain match
+  (`docs.python.org` covered by `python.org`); `www.` stripped. Opt
+  out with `new NullPii({ urlAllowlist: 'none' })`.
+
+  An allowlisted URL is dropped **only when it carries no other PII
+  span**. `core:url` matches `[^\s<>"]+`, so PII glued to a URL is
+  swallowed into the URL span and containment-elimination then drops
+  the inner span regardless of score; dropping the outer span would
+  have emitted the nested secret as plaintext behind a 20-character
+  allowlisted prefix (`https://github.com/a,AKIA…`). Fail-safe by
+  construction: a reference URL embedding anything identifying reverts
+  to whole-URL redaction. Also refuses to judge a span carrying more
+  than one scheme (`core:url` does not stop at `,`), and matches on
+  `hostname` rather than `host` so a non-default port no longer
+  silently disables the allowlist.
+
+### Dropped before release
+
+- **Template syntax mask** (#44) — **closed, not shipped.** #43 already
+  fixes the motivating case: ten template-variable probes on `main`,
+  including the two this feature cited, produce zero tagged spans, and
+  the regression test in `test/nullpii.test.ts` passes without it. The
+  implementation also made `MAX_INPUT_BYTES`-sized input quadratic
+  (69.7 s on 1 MB, against ~70 ms for the whole rest of the pipeline),
+  and unbalanced delimiters suppressed redaction in ordinary text — a C
+  array initializer and a one-character typo both leaked real emails.
+- **`private_date` threshold 0.85** (#46) — **closed, not shipped.** It
+  does not reach its own target: gateway boilerplate dates score
+  0.993–1.000, so an 0.85 cut removes none of them. The score tracks
+  span-boundary confidence, not PII relevance — a birth date inside a
+  two-date span scores 0.574 and a neutral one scores 0.609 — so no cut
+  point separates them. Measured cost was **419 true positives dropped
+  to remove 453 false ones** (1.08 : 1), with `private_date` F1 going
+  *down* 0.004. What it did remove was boundary fragments (`'62'`, a
+  bare `'1938'` split off `March 17, 1938`), not the copyright footers
+  it was aimed at.
+
+### Bench
+
+The benchmark's ai4privacy and isotonic loaders resolved unknown
+upstream labels through `dict.get()`, which returns `None` both for
+"deliberately excluded" and for "never seen". Gold was deleted with no
+signal — **56.4%** on `ai4privacy-300k-heldout`, ~11% on the isotonic
+`-heldout` rows. Because `macro_f1` skips zero-support classes,
+predictions on a class whose gold had been deleted cost nothing, so the
+bug hid recall *and* inflated precision. All 25 recovered labels are
+aliases of keys already in the maps; mapping and exclusion are now
+distinct and an unknown label raises.
+
+This moves every tool, not just `nullpii` — on `ai4privacy-300k-heldout`
+it reorders 8 of 9 (`piiranha` +0.2347, `gliner-pii-large-v1` +0.1801,
+`gliner-onnx-pii-fp32` +0.1423, `nullpii` −0.0519, `presidio` −0.1746).
+The published v0.3.0 column is therefore **not comparable** to the
+v0.4.0 column; the middle column restates v0.3.0 on the corrected
+metric, and only the last delta is a product improvement.
+
+| Metric | v0.3.0 as published | v0.3.0 re-scored | v0.4.0 | Δ (product) |
+|---|---:|---:|---:|---:|
+| OOD-7 macro F1 | 0.7784 | 0.8043 | **0.8290** | **+0.0247** |
+| `nullpii-bench` F1 | 0.4228 | 0.4228 | **0.4519** | **+0.0291** |
+| Cold start (M5 Pro CPU) | ~756 ms | — | not re-measured | — |
+
+`presidio-synthetic` moves by exactly `+0.0000` — it carries no
+`private_geolocation` gold and none of its 89 URLs are allowlisted,
+which makes it the negative control for both behavior changes.
+
+The 11 canonical rows the gold fix did not touch still carry v0.3.0
+numbers; the published table is not fully regenerated yet.
+
+### Config additions
+
+- `NullPiiConfig.urlAllowlist?: 'none'` — opt out of public-URL allowlist.
+
+### Internal
+
+- `src/geo-filter.ts` — `isCoordinateShaped`, `dropNonCoordinateGeolocation`. Range and null-island rejection defer to the existing `latLonPairInRange`.
+- `src/url-filter.ts` — `PUBLIC_URL_HOSTS`, `isPublicUrl`, `dropCleanPublicUrlSpans`.
+- `src/placeholder-escape.ts` — sentinel chars now exported individually as `PLACEHOLDER_SENTINEL_LEFT` / `PLACEHOLDER_SENTINEL_RIGHT` for reuse in `normalize.ts`. Source uses `''` / `''` escape syntax instead of raw PUA (grep-friendly, no mojibake on GitHub mobile).
+- `packages/eval/scripts/ood_macro.py` — single definition of the OOD-7 set, computed from any `matrix.json`. The root README enumerated 7 datasets and `packages/eval/datasets/README.md` said 5, a 0.0128 discrepancy that would have silently blocked any regression gate. Exits non-zero if a cell is missing or `CRASHED`.
+- `packages/eval/pyproject.toml` — the wheel force-included the whole `datasets/` directory, so any file dropped there was redistributed. Now a per-file allowlist with the upstream licence annotated per line.
+
+### Known issues
+
+- Partially-overlapping spans corrupt the output and break the restore
+  round-trip. `x https://github.com/users/john.doe@acme.com y` yields
+  `private_url[2,44]` and `private_email[27,46]` — IoU 0.39, below the
+  dedupe threshold, so both survive and overwrite each other during
+  vault substitution; the email span also ends past the end of the
+  string. Pre-existing in 0.3.0, not introduced here.
+
+### Test plan
+
+- 300 unit + integration tests passing (was 271 on v0.3.0).
+- New tests: `test/geo-filter.test.ts` (10 — coordinate shapes, the place names and postcodes the head mislabels, IPv4 not reading as a pair, range and null-island rejection), `test/url-filter-nested.test.ts` (10 — nested-secret retention, concatenated URLs, non-default ports, near-miss hosts), `test/url-filter.test.ts` (7), plus integration cases in `test/nullpii.test.ts`.
+- `test/url-filter.test.ts`'s span fixture was corrected: it parked every span at offset 0, which under a containment-aware drop rule reads as "this URL has PII nested inside it".
+
 ## [0.3.0] — first public release
 
 Initial Apache-2.0 public release. Published packages:

README.md

@@ -150,7 +150,7 @@ Two `nullpii` rows + one upstream-GLiNER row let readers isolate the model from
 - **`gliner-onnx-pii-fp32`** — the unmodified upstream [`onnx-community/gliner_multi_pii-v1`](https://huggingface.co/onnx-community/gliner_multi_pii-v1) ONNX, same bare consumer. Baseline before any project fine-tuning.
 - **`nullpii`** — the npm package (full runtime): published model + recognizer pack + adversarial preprocessor + base64 decoder + reversible vault.
 
-v0.3.0 bench (M5 Pro CPU, 2026-05-18 + opf 2026-05-20, full 9×16 matrix). OOD macro for `nullpii` = **0.7784** (presidio-synthetic + isotonic-{en,de,fr,it}-heldout + ai4privacy-300k-heldout + tab-echr).
+v0.3.0 bench (M5 Pro CPU, 2026-05-18 + opf 2026-05-20, full 9×16 matrix). OOD macro for `nullpii` = **0.7784** over the **OOD-7** set (presidio-synthetic + isotonic-{en,de,fr,it}-heldout + ai4privacy-300k-heldout + tab-echr). The set is defined once, in `packages/eval/scripts/ood_macro.py`; recompute the headline from any `matrix.json` with `python packages/eval/scripts/ood_macro.py <matrix.json>` rather than restating it by hand.
 
 | Dataset | n | **`nullpii`** | **`nullpii-bare`** | `nemotron-pii-raw` | `gliner-pii-large-v1` | `gliner-onnx-pii-fp32` | `deberta` | `piiranha` | `presidio` | `opf` |
 |---|---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
@@ -174,7 +174,8 @@ v0.3.0 bench (M5 Pro CPU, 2026-05-18 + opf 2026-05-20, full 9×16 matrix). OOD m
 Legend:
 - **bold** = best F1 in the row
 - ⚠ = the dataset overlaps the training distribution of at least one competitor in the row — read those cells with caution
-- ⚐ = in-distribution for `nullpii` itself — regression cell, **not** counted in the OOD headline. The held-out OOD macro (0.7784) is computed over `presidio-synthetic` + `isotonic-{en,de,fr,it}-heldout` + `ai4privacy-300k-heldout` + `tab-echr` only. The `nullpii-internal-bench` row sits at the bottom of the table and is shown only as a regression watcher across releases — read it that way.
+- ⚐ = in-distribution for `nullpii` itself — regression cell, **not** counted in the OOD headline. The held-out OOD macro (0.7784) is computed over the OOD-7 set only: `presidio-synthetic` + `isotonic-{en,de,fr,it}-heldout` + `ai4privacy-300k-heldout` + `tab-echr`. The `nullpii-internal-bench` row sits at the bottom of the table and is shown only as a regression watcher across releases — read it that way.
+- the `-heldout` suffix means rows sliced **above `nullpii`'s own training offsets** (`_AI4_HELDOUT_OFFSET` / `_ISOTONIC_HELDOUT_ROW_OFFSET` in `packages/eval/scripts/bench_full.py`). It is a guarantee about `nullpii` and about no other tool in the row — a competitor trained on the same upstream corpus sees those rows as ordinary training data.
 - ‡ = competitor benched on its own training distribution (best-case self-report)
 - § = Presidio benched on its own evaluator dataset (best-case self-report)
 

packages/eval/datasets/README.md

@@ -10,8 +10,14 @@ externally-licensed upstream splits.
 | File | Rows | Used in canonical bench | Origin |
 |---|---:|:---:|---|
 | `nullpii-bench.jsonl` | 2,361 | ✅ project-authored, see subsets below | self-authored, Apache-2.0 |
-| `presidio-synthetic.jsonl` | 5,000 | ✅ external (held-out OOD-5) | Microsoft Presidio seed, MIT |
-| `tab-echr-test.jsonl` | 127 | ✅ EU legal test split | TAB ECHR test (ACL 2022), MIT |
+| `presidio-synthetic.jsonl` | 5,000 | ✅ external, member of **OOD-7** | Microsoft Presidio seed, MIT |
+| `tab-echr-test.jsonl` | 127 | ✅ EU legal test split, member of **OOD-7** | TAB ECHR test (ACL 2022), MIT |
+
+Every file listed here is redistributed inside the `nullpii-eval` wheel via
+the per-file allowlist in [`../pyproject.toml`](../pyproject.toml). A file
+dropped into this directory does **not** ship until it is added there —
+which is the point: adding the line forces a licence check first. Only
+MIT / Apache-2.0 / BSD / ISC / CC0 upstreams are eligible.
 
 Additional bench rows (`ai4privacy-*`, `isotonic-*`, `nemotron-pii-*`,
 `argilla-pii`) are fetched from HuggingFace at bench time by the per-tool
@@ -64,10 +70,38 @@ re-fetching the upstream pool from internal storage.
 
 ⚠ `nullpii-bench` is in-distribution for the project pipeline. F1 on
 this dataset is a regression test for the runtime (preprocessor,
-recognizer pack, base64 decoder), not an OOD generalisation claim. The
-held-out OOD headline is the macro over 5 external datasets
-(`presidio-synthetic` + `isotonic-{en,de,fr,it}-heldout`) reported in
-the top-level README.
+recognizer pack, base64 decoder), not an OOD generalisation claim.
+
+### OOD-7 — the held-out headline set
+
+The OOD headline reported in the top-level README is the macro over
+**seven** external datasets, named **OOD-7** here and in the root README:
+
+```
+presidio-synthetic
+isotonic-en-heldout   isotonic-de-heldout
+isotonic-fr-heldout   isotonic-it-heldout
+ai4privacy-300k-heldout
+tab-echr
+```
+
+Membership criterion: the dataset is externally authored **and** no part
+of it entered nullpii's training distribution. The `-heldout` suffix means
+rows sliced above nullpii's own training offsets
+(`_AI4_HELDOUT_OFFSET` / `_ISOTONIC_HELDOUT_ROW_OFFSET`,
+[`../scripts/bench_full.py`](../scripts/bench_full.py)) — it carries **no**
+held-out guarantee for any other tool in the matrix.
+
+Do not restate this number by hand. Compute it from `matrix.json`:
+
+```bash
+python packages/eval/scripts/ood_macro.py packages/eval/published-bench/matrix.json
+```
+
+An earlier revision of this file described the headline as a macro over
+five datasets; that was wrong and disagreed with the root README by
+0.0128 F1 (OOD-5 = 0.7656 vs OOD-7 = 0.7784 at v0.3.0). The root README
+figure was the correct one.
 
 ## `presidio-synthetic`
 

packages/eval/pyproject.toml

@@ -22,5 +22,15 @@ build-backend = "hatchling.build"
 [tool.hatch.build.targets.wheel]
 packages = ["src/nullpii_eval"]
 
+# Per-file allowlist, NOT a directory include. Anything under `datasets/`
+# that is not named here stays out of the distributed Apache-2.0 wheel.
+# Adding a line is a deliberate act that requires checking the upstream
+# licence first: this directory is packaged and redistributed, so a file
+# with incompatible terms (anything outside MIT/Apache-2.0/BSD/ISC/CC0)
+# must never be listed. Bench rows fetched from HuggingFace at run time
+# are unaffected — they are never written here.
 [tool.hatch.build.targets.wheel.force-include]
-"datasets" = "nullpii_eval/datasets"
+"datasets/README.md" = "nullpii_eval/datasets/README.md"
+"datasets/nullpii-bench.jsonl" = "nullpii_eval/datasets/nullpii-bench.jsonl"          # Apache-2.0, project-authored
+"datasets/presidio-synthetic.jsonl" = "nullpii_eval/datasets/presidio-synthetic.jsonl"  # MIT, Microsoft Presidio
+"datasets/tab-echr-test.jsonl" = "nullpii_eval/datasets/tab-echr-test.jsonl"          # MIT, TAB ECHR (ACL 2022)

packages/eval/scripts/ood_macro.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python3
+"""Compute the OOD-7 headline macro-F1 from a bench `matrix.json`.
+
+The held-out OOD headline quoted in the top-level README used to be a
+hand-maintained number. Nothing computed it, so the root README and
+`packages/eval/datasets/README.md` drifted apart: the root enumerated
+seven datasets (0.7784) while the datasets README claimed five (0.7656),
+a 0.0128 gap that silently invalidated any regression gate measured
+against the wrong baseline.
+
+This script is the single source of truth. `OOD_7` below is the set;
+change it here and nowhere else.
+
+Membership criterion: externally authored AND no part of it entered
+nullpii's training distribution. The `-heldout` suffix means rows sliced
+above nullpii's own training offsets (`_AI4_HELDOUT_OFFSET` /
+`_ISOTONIC_HELDOUT_ROW_OFFSET` in `bench_full.py`) — it carries no
+held-out guarantee for any other tool in the matrix.
+
+Usage:
+  python ood_macro.py packages/eval/published-bench/matrix.json
+  python ood_macro.py matrix.json --tool nullpii-bare
+  python ood_macro.py a/matrix.json --baseline b/matrix.json   # A/B delta
+
+Exits non-zero if any OOD-7 cell is missing or did not complete, so a
+truncated bench run can never be reported as a headline.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+# The held-out OOD headline set. Order is the reporting order.
+OOD_7 = (
+    "presidio-synthetic",
+    "isotonic-en-heldout",
+    "isotonic-de-heldout",
+    "isotonic-fr-heldout",
+    "isotonic-it-heldout",
+    "ai4privacy-300k-heldout",
+    "tab-echr",
+)
+
+_OK_STATUS = "OK"
+
+
+def ood_cells(matrix: dict, tool: str) -> dict[str, float]:
+    """Per-dataset F1 for `tool` across OOD-7. Raises if any cell is unusable."""
+    cells: dict[str, float] = {}
+    missing: list[str] = []
+    for key in OOD_7:
+        cell = matrix.get(key, {}).get(tool)
+        if cell is None:
+            missing.append(f"{key}: absent")
+            continue
+        status = cell.get("status", _OK_STATUS)
+        if status != _OK_STATUS:
+            missing.append(f"{key}: status={status}")
+            continue
+        f1 = cell.get("f1")
+        if f1 is None:
+            missing.append(f"{key}: no f1")
+            continue
+        cells[key] = float(f1)
+    if missing:
+        raise SystemExit(
+            f"OOD-7 incomplete for tool '{tool}' — refusing to report a headline:\n  "
+            + "\n  ".join(missing)
+        )
+    return cells
+
+
+def ood_macro(matrix: dict, tool: str) -> float:
+    """Unweighted mean of the seven OOD-7 per-dataset macro-F1 scores."""
+    cells = ood_cells(matrix, tool)
+    return sum(cells.values()) / len(cells)
+
+
+def _load(path: Path) -> dict:
+    if not path.exists():
+        raise SystemExit(f"no such matrix: {path}")
+    return json.loads(path.read_text())
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("matrix", type=Path, help="path to matrix.json")
+    ap.add_argument("--tool", default="nullpii", help="tool column (default: nullpii)")
+    ap.add_argument("--baseline", type=Path, default=None,
+                    help="second matrix.json to diff against (A/B mode)")
+    args = ap.parse_args()
+
+    cells = ood_cells(_load(args.matrix), args.tool)
+    macro = sum(cells.values()) / len(cells)
+
+    if args.baseline is None:
+        for key, f1 in cells.items():
+            print(f"  {key:26} {f1:.4f}")
+        print(f"\nOOD-7 macro [{args.tool}] = {macro:.4f}  (n={len(cells)} datasets)")
+        return
+
+    base = ood_cells(_load(args.baseline), args.tool)
+    base_macro = sum(base.values()) / len(base)
+    print(f"  {'dataset':26} {'baseline':>9} {'candidate':>10} {'delta':>9}")
+    for key in OOD_7:
+        delta = cells[key] - base[key]
+        flag = "  " if abs(delta) < 5e-4 else (" +" if delta > 0 else " -")
+        print(f"  {key:26} {base[key]:9.4f} {cells[key]:10.4f} {delta:+9.4f}{flag}")
+    delta = macro - base_macro
+    print(f"\n  {'OOD-7 macro':26} {base_macro:9.4f} {macro:10.4f} {delta:+9.4f}")
+    if delta < 0:
+        print("\nREGRESSION — candidate is worse than baseline.", file=sys.stderr)
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()