Extract shared materializer helpers; make benchmark-specific scoring/policy data-driven

[shanev]

Context

Code review across the #166 benchmark PRs (S-NIAH #170, LongBench-v2 CodeQA #171, OOLONG-Pairs #169) surfaced a consistent, growing pattern of technical debt as more benchmark families get added:

1. Materializer duplication (rule of three, now hit). _encode_json, _write_exclusive (atomic no-overwrite write via tempfile+fsync+hardlink), and the HTTP-download-with-User-Agent-and-error-wrap pattern are copy-pasted byte-for-byte across benchmarks/oolong.py, benchmarks/sniah.py, and benchmarks/longbench_codeqa.py (and presumably benchmarks/oolong_pairs.py). This is the atomic-write contract CLAUDE.md documents as load-bearing for the immutable-artifact guarantee — a future fix to it needs applying in every copy or it silently diverges. Extract into shared infrastructure (e.g. benchmarks/_materialize.py) with encode_json, write_exclusive, and a parameterized download helper.

2. _policy_for_task in benchmarks/live.py accretes a hardcoded if benchmark_id == "..." branch per benchmark. Now 3 branches (OOLONG task-shape fallback, s-niah, longbench-v2-codeqa); 4 more benchmarks are already planned in the manifest (BrowseComp-Plus, OOLONG-Pairs, TAG-Bench, +1). BenchmarkSpec in benchmarks/models.py has no field for per-benchmark guidance, so there's no data-driven home for it — every new benchmark means another elif in shared infrastructure. Consider declaring semantic_policy/guidance on the manifest's benchmark entry so _policy_for_task looks it up instead of growing forever.

3. exact_match scorer sniffs domain shape. benchmarks/scoring.py's exact_match now branches on expected in {"a","b","c","d"} to apply multiple-choice-specific leniency (added for LongBench-v2 CodeQA), mixing implicit domain logic into a scorer 3+ benchmarks depend on. A future benchmark whose real answer happens to be a bare 'a'-'d' character would silently inherit letter-extraction semantics it never asked for. Consider a distinct multiple_choice ScorerKind declared explicitly in the manifest instead.

4. Manifest snapshot duplication. Per-run manifest snapshots (rlm-paper-v1-oolong-2026-07-17.json, -sniah-, -longbench-v2-codeqa-) each re-embed full arm/model/limits blocks rather than referencing a shared definition, and which sibling benchmarks each snapshot marks "ready" is inconsistent across branches (undocumented, easy to get wrong when a reader materializes only what they think they need).

Not blocking

None of these are correctness bugs in the current, merged state — all are flagged PLAUSIBLE/cleanup-severity in review, not CONFIRMED bugs. This is deferred cleanup, not a blocker for #166's remaining merges.

[shanev]

Additional findings from BrowseComp-Plus's code review (not yet merged, real run still in progress):

• The atomic-write duplication now extends into benchmarks/live.py too (`_write_context_cache`, a 5th independent copy), outside this issue's original "every materializer" scope.
• `_is_modelrelay_oversized_payload_error()` (benchmarks/live.py) classifies ANY ModelRelay 400 request-layer validation error matching a generic shape as `context_limit`, not specifically size-related rejections — the signature carries zero size-specific information. Currently correct in practice (uniform, consistent across all 150 real direct-arm BrowseComp-Plus attempts, and contextually obvious given 24-44MB payloads vs any model's window), but the mechanism is a heuristic pattern-match on a JSON body that `http_error_excerpt()` truncates to 300 chars — already observed truncating mid-object in production (a committed LongBench artifact). A future ModelRelay error-envelope change could silently defeat the classifier with no visible failure. Consider a genuine pre-flight payload-size check against the model's known context window as a more robust, independent signal, rather than relying solely on pattern-matching the server's generic rejection.
• `render_document()` (benchmarks/browsecomp_plus.py) calls `.get()` on unvalidated decrypted list items without checking they're dicts — a malformed/non-dict entry crashes with a raw AttributeError instead of the intended `BenchmarkRunError`. Low-probability (would require a corrupted/unusual dataset row), but worth hardening for defense-in-depth.
• `_policy_for_task`'s hardcoded per-benchmark branch pattern (already tracked here) grew a 4th case for browsecomp-plus-1k — confirms the debt is compounding with every new benchmark, no new issue needed.