Mlin/reduce hardcoded heuristics

AGENTS.md

@@ -10,6 +10,9 @@ Violations will cause runs to be blocked or reverted and large multimillion doll
 * Strictly follow existing style in the codebase.
 * Ensure every function is properly documented following the existing style.
 * Write in-line comments strategically, especially for key business logic. Do this sparingly, and strategically.
+* DEVELOP A STRONG DISTASTE FOR REGEX/TEXT HEURISTICS.
+    - Favor using established external libraries instead
+    - Favor using proper, generalized functions
 * Keep code concise, tasteful and elegant.
 * Don't overcomplicate things. Don't implement more than you need to.
 * Ensure code is readable by humans, easy to extend and maintain.
@@ -82,6 +85,11 @@ you must ensure those comments continue to exist in the new/migrated function/co
     including modifications to `LOGBOOK.md`, `agent_logs/`, `CHANGELOG.md` and so on.
     - NEVER undo others' work.
     - Keep calm and continue executing with your plan. You do not need to stop and ask me about it.
+* WHEN WRITING UNIT TESTS:
+    - Monkeypatching is fine for isolating specific logic to test for, but you should NOT abuse it.
+    - You should still have proper "integration" style tests that pass in legitimate production-style inputs, and assert for legitimate outputs (as far as pragmatically possible).
+    - When dealing with tests that require external APIs like LLM APIs, of course monkeypatching helps to run these tests locally without actually calling the external LLM API and incurring costs. However, we should still set aside a comprehensive suite of integration tests that actually make the required LLM calls, and do comprehensive asserts/checks on the outputs. These tests can be run in special environment with LLM key available, and can be disabled by default for normal CI tests.
+    - But these tests should still exist.
 
 ## Testing rules
 

BENCHMARK_REDUCED_HEURISTICS.md

@@ -0,0 +1,142 @@
+# Benchmark: Reduced-Heuristics Branch
+
+_Last updated: 2026-02-18_
+
+## 1) Scope
+This report covers Task 2 for branch `mlin/reduce-hardcoded-heuristics`:
+- re-run the eval pipeline with current deploy-matched settings,
+- analyze failures,
+- perform a manual judge audit (Codex reasoning, no judge-LLM self-audit),
+- compare against previously recorded benchmark baselines in `BENCHMARK.md`.
+
+## 2) Run Configuration
+Core settings used:
+- mode: `normal`
+- tools: enabled
+- refine: disabled
+- generation workers: `12` (thread backend)
+- query timeout/retries: `350s`, `1`
+- judge workers: `12`
+- judge context: `80000`
+- judge timeout/retries: `350s`, `1`
+- schema: `eval_revamp_combined_512_20260217`
+
+Run group manifest:
+- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.manifest.json`
+
+Run dirs:
+- `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.single100.normal.tools12.norefine.20260218_195034`
+- `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.multi60.normal.tools12.norefine.20260218_200838`
+- `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.open200.normal.tools12.norefine.20260218_202301`
+
+## 3) Topline Metrics (This Run)
+
+### 3.1 Generation throughput/latency
+| suite | n | n_err | avg_total_ms | wall_total_ms | qps |
+|---|---:|---:|---:|---:|---:|
+| single100 | 100 | 1 | 59,540.45 | 846,854.45 | 0.1181 |
+| multi60 | 60 | 0 | 134,343.77 | 708,687.42 | 0.0847 |
+| open200 | 200 | 0 | 63,161.56 | 1,074,835.48 | 0.1861 |
+
+### 3.2 Judge-facing fail rates
+- single100 (`score_summary.json`):
+  - factual fail: `0.0882`
+  - open faithfulness fail: `0.1333`
+  - refusal fail: `0.0000`
+  - distractor focus fail: `0.0667`
+- multi60:
+  - comparison fail: `0.1167`
+- open200:
+  - open faithfulness fail: `0.1350`
+  - open helpfulness fail: `0.0050`
+
+## 4) Comparison vs Previous Baseline (`BENCHMARK.md`)
+Reference row in `BENCHMARK.md`:
+- `baseline_normal`: factual fail `0.0857`, open faith fail `0.1667`, comparison fail `0.0167`.
+
+Comparison (closest axes):
+- factual fail: `0.0882` (near parity; slightly worse by +0.0025)
+- open faithfulness fail: `0.1333` (improved by -0.0334)
+- comparison fail: `0.1167` (material regression, +0.1000)
+
+Interpretation:
+- removing brittle heuristics did **not** materially hurt factual fail rate,
+- faithfulness on open-ended remained improved versus historical baseline,
+- comparison handling regressed strongly and is now the dominant quality gap on multi-ticker prompts.
+
+## 5) Timeout Incident Log (Generation)
+Observed and captured in `agent_logs/LOGBOOK.md`:
+- hard failure query:
+  - `query_id=1dd6251b-e62b-4e58-ae52-35a1253e14c3`
+  - question: "What was LITE's net income in its 10-Q filed 2026-02-04?"
+  - failure: timed out after 2 attempts (`350s` + retry), `n_err=1`
+  - scavenged output: no draft/final answer, no tool trace, generation error record persisted.
+- recovered long-tail examples:
+  - `aada22de-6020-41aa-be15-5516f64b0aca` (MSFT total revenue) succeeded on retry.
+  - `598beb04-ec0c-4314-893e-2deb8f167179` (INTC vs NVDA comparison) succeeded on retry.
+
+### 5.1 Isolated replay of the LITE timeout query
+To check whether this was purely batch-queue starvation, I ran the same query in isolation.
+
+- Query: `What was LITE's net income in its 10-Q filed 2026-02-04?`
+- Probe A (`agent_logs/scripts/eval/20260218_220200_probe_lite_isolated_latency.sh`):
+  - direct single-call runtime probe with outer `timeout 500s`
+  - outcome: process timed out (`exit 124`) before returning
+- Probe B (`agent_logs/scripts/eval/20260218_221400_probe_lite_single_eval_timeout350.sh`):
+  - single-query `run_eval` with `concurrency=1`, `query_timeout_s=350`, `query_max_retries=0`
+  - run dir: `agent_logs/reports/retrieval_eval_20260218/lite_single_eval_probe/eval_run.lite_isolated_timeout350.20260218_220015`
+  - outcome: success in `20601 ms` (`n_ok=1`, `n_err=0`), with tool trace and retrieved/reranked chunks present.
+
+Inference:
+- The long-tail timeout is not only a batching artifact; isolated calls can still hit pathological slow behavior.
+- However, the same query can also complete quickly in isolated eval mode, which is consistent with intermittent decode/runtime stalls rather than deterministic query complexity.
+
+## 6) Manual Judge Audit (Codex, Non-Circular)
+
+### 6.1 Method
+To avoid circularity, the audit did **not** call the judge LLM.
+- Built decision table:
+  - `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.judge_audit_manual/decision_audit.raw.csv`
+- Full 698-row decision set was split into six shards and manually labeled by Codex workers using rubric-by-`judge_id` reasoning.
+- Merged labeled output:
+  - `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.judge_audit_manual/decision_audit.codex_manual.csv`
+- Reliability report:
+  - `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.judge_audit_manual/judge_reliability_report.codex_manual.json`
+
+### 6.2 Alignment summary (test split)
+| judge | n_test | accuracy | precision_fail | recall_fail | notes |
+|---|---:|---:|---:|---:|---|
+| faithfulness_v1 | 58 | 0.9828 | 0.8750 | 1.0000 | strong alignment |
+| factual_correctness_v1 | 9 | 1.0000 | 1.0000 | 1.0000 | tiny sample |
+| helpfulness_v1 | 85 | 0.9882 | 1.0000 | 0.5000 | under-calls fail cases |
+| comparison_v1 | 15 | 1.0000 | 1.0000 | 1.0000 | aligned on this set |
+| focus_v1 | 4 | 1.0000 | 0.0000 | 0.0000 | no fail cases in test split |
+| refusal_v1 | 5 | 0.8000 | 0.0000 | 0.0000 | misses refusal-needed cases |
+
+### 6.3 Key disagreement patterns
+Confusion from full 698 labeled decisions:
+- false positives: `4` total
+  - mostly faithfulness over-flags (`3`) and one factual false positive.
+- false negatives: `10` total
+  - helpfulness under-flags (`6`) for non-responsive comparison/analysis answers,
+  - refusal under-flags (`3`) where out-of-scope prompts were met with clarification instead of refusal,
+  - faithfulness under-flag (`1`) on unsupported filing-availability claim.
+
+## 7) Genuine Pipeline Failures (from manual audit)
+Main categories of true failures (`human_label=1`):
+- comparison completeness failures (`comparison_v1`, `7`): model defers/clarifies instead of producing requested side-by-side analysis.
+- open-ended faithfulness failures (`faithfulness_v1`, `29`): period mismatch and unsupported specific claims remain the largest category.
+- refusal behavior gaps (`refusal_v1`, `3`): out-of-scope ticker prompts not refused strongly enough.
+- helpfulness failures (`helpfulness_v1`, `8`): mostly non-answers/deferrals for requested comparative analysis.
+
+## 8) Surprising Findings and Hypotheses
+1. Removing brittle heuristics improved maintainability without collapsing factual/open-ended quality.
+2. Multi-ticker comparison degraded sharply; likely because previous heuristic scaffolding implicitly forced comparative structure.
+3. Faithfulness judge quality is now relatively strong under Codex-manual audit; biggest remaining reliability issue is helpfulness/refusal under-calling.
+4. Timeout outliers still materially affect wall-clock and can dominate throughput for small suites.
+
+## 9) Immediate Follow-ups
+1. Improve comparison answer planning (explicit required-output structure for multi-ticker comparison prompts).
+2. Tighten refusal policy for out-of-scope tickers (prefer explicit refusal over vague clarifying loops).
+3. Add retry+continue safeguards for long-tail timed-out generations (already partially in place).
+4. Keep judge audits separated from judge-model outputs (Codex-manual process retained as the non-circular check).

BENCHMARK_RETRIEVAL.md

@@ -0,0 +1,178 @@
+# Benchmark: Retrieval + Rerank Quality (Reduced-Heuristics Branch)
+
+## 1) Scope
+This report continues from the completed 300-sample manual retrieval audit and closes the retrieval-focused evaluation workstream.
+
+Primary goals:
+- quantify retriever vs reranker behavior,
+- quantify evidence-support quality on open-ended answers,
+- audit retrieval relevance with non-circular manual labels (Codex reasoning, not judge-LLM),
+- calibrate weak labels against manual labels.
+
+All runs below use the same reduced-heuristics full-suite manifest:
+- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.manifest.json`
+
+## 2) Experiments Run
+
+| ID | Experiment | Input artifacts | Output artifacts | What it measures |
+|---|---|---|---|---|
+| E1 | Factual retrieval/rerank IR metrics (`single100`) | `eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.single100.normal.tools12.norefine.20260218_195034` | `retrieval_rerank_metrics.json`, `retrieval_rerank_metrics.csv`, `retrieval_nli_claim_support.csv` | Pre vs post rerank MRR/hit/precision/recall using factual gold evidence anchors |
+| E2 | Open-ended NLI evidence support (`open200`) | `eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.open200.normal.tools12.norefine.20260218_202301` | `retrieval_rerank_metrics.json`, `retrieval_nli_claim_support.csv` | Claim support / contradiction / unsupported rates |
+| E3 | Multi slice retrieval pass (`multi60`) | `eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.multi60.normal.tools12.norefine.20260218_200838` | `retrieval_rerank_metrics.json` | Completeness check for full-suite parity (no factual/open-ended rows in this slice) |
+| E4 | Retrieval candidate pool build | all three run dirs above | `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.retrieval_pool.csv`, `.stats.json` | pooled chunk candidates for manual relevance auditing |
+| E5 | Manual relevance audit (300 rows) | `...retrieval_pool.sample300.csv` | `...retrieval_pool.sample300.codex_manual.csv` | Human-proxy relevance labels via Codex reasoning (non-circular) |
+| E6 | Weak-label calibration | `...sample300.codex_manual.csv` | `...sample300.calibration.json` | Weak-label precision/recall/alignment vs manual labels |
+| E7 | Manual audit summary rollup | `...sample300.codex_manual.csv` | `agent_logs/reports/retrieval_eval_20260218/manual_sample300_summary.json`, `.md` | relevance prevalence, pre/post membership, rank movement, top-k relevance slices |
+
+## 3) Core Results
+
+### 3.1 Retriever vs reranker on factual anchors (E1)
+
+| Metric | Pre-rerank | Post-rerank | Delta |
+|---|---:|---:|---:|
+| factual_n | 34 | 34 | - |
+| chunk MRR | 0.3092 | 0.1743 | -0.1349 |
+| chunk win rate | - | - | 0.1765 |
+| chunk precision@5 | 0.1118 | 0.0647 | -0.0471 |
+| chunk precision@10 | 0.0647 | 0.0471 | -0.0176 |
+| chunk precision@25 | 0.0294 | 0.0282 | -0.0012 |
+| chunk recall@25 | 0.7353 | 0.7059 | -0.0294 |
+| doc MRR | 1.0000 | 1.0000 | 0.0000 |
+
+Interpretation:
+- On factual gold-anchor queries, current reranking is net negative on chunk-level relevance concentration.
+- Doc-level MRR is saturated at 1.0 and is not discriminative for this run.
+
+### 3.2 NLI claim support on open-ended generations (E1/E2)
+
+| Slice | n_open_ended_scored | support_rate | contradiction_rate | unsupported_rate |
+|---|---:|---:|---:|---:|
+| `single100` subset | 30 | 0.1000 | 0.3958 | 0.5042 |
+| `open200` | 120 | 0.1292 | 0.4115 | 0.4594 |
+
+Interpretation:
+- Support is low and contradiction/unsupported are high.
+- This aligns directionally with remaining faithfulness pressure points in answer-level evals.
+
+### 3.3 Multi slice parity check (E3)
+
+`multi60` has no factual or open-ended rows, so retrieval IR/NLI outputs are expectedly `NaN`/empty for these specific metric families.
+
+## 4) Manual 300-Sample Relevance Audit (E5/E7)
+
+Manual labels are from Codex reasoning on each row, not from the judge LLM.
+
+Source:
+- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.codex_manual.csv`
+
+Summary:
+- `n_labeled = 300`
+- overall positive relevance rate = `0.4167`
+
+By query kind:
+
+| Kind | n | n_positive | positive_rate |
+|---|---:|---:|---:|
+| factual | 140 | 19 | 0.1357 |
+| open_ended | 90 | 53 | 0.5889 |
+| comparison | 55 | 44 | 0.8000 |
+| distractor | 15 | 9 | 0.6000 |
+
+Pre/post membership buckets:
+
+| Bucket | n | n_positive | positive_rate |
+|---|---:|---:|---:|
+| both pre+post | 195 | 74 | 0.3795 |
+| pre-only | 53 | 26 | 0.4906 |
+| post-only | 52 | 25 | 0.4808 |
+
+Relevant-rank movement (rows present in both pre and post, relevance=1):
+- n=74, promoted=32, demoted=37, unchanged=5, avg delta(post-pre)=+0.0676
+
+By kind (same movement view):
+
+| Kind | n | promoted | demoted | unchanged | avg_delta(post-pre) |
+|---|---:|---:|---:|---:|---:|
+| factual | 15 | 4 | 9 | 2 | +2.8000 |
+| open_ended | 38 | 19 | 17 | 2 | -1.8421 |
+| comparison | 17 | 6 | 10 | 1 | +3.4118 |
+| distractor | 4 | 3 | 1 | 0 | -6.2500 |
+
+Sample top-k relevance slices (not an unbiased absolute P@k estimator; useful as directional diagnostics):
+
+| Phase | k | n_rows_in_slice | n_positive | positive_rate |
+|---|---:|---:|---:|---:|
+| pre | 5 | 63 | 40 | 0.6349 |
+| post | 5 | 64 | 37 | 0.5781 |
+| pre | 10 | 128 | 60 | 0.4688 |
+| post | 10 | 134 | 71 | 0.5299 |
+
+Interpretation:
+- At very early ranks (top-5), audited relevance is lower post-rerank than pre-rerank.
+- At top-10, post-rerank recovers and slightly exceeds pre-rerank in this sample.
+- Factual and comparison rows show more demotions than promotions, consistent with E1 factual-anchor degradation.
+
+## 5) Weak-Label Calibration (E6)
+
+Source:
+- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.calibration.json`
+
+Threshold: weak relevance >= 0.5 => relevant.
+
+| Metric | Value |
+|---|---:|
+| n | 140 |
+| accuracy | 0.1357 |
+| precision_1 | 0.1357 |
+| recall_1 | 1.0000 |
+| f1_1 | 0.2390 |
+| balanced_accuracy | 0.5000 |
+| tp / fp / tn / fn | 19 / 121 / 0 / 0 |
+
+Interpretation:
+- Current weak labels are recall-maximal but extremely low precision (many false positives).
+- They are suitable as high-recall candidate generation tags, not as ground-truth proxies for precision-sensitive decisions.
+
+## 6) Surprising Findings and Hypotheses
+
+1. Reranking currently hurts factual chunk concentration.
+- Evidence: negative chunk MRR and precision deltas on factual anchors.
+- Hypothesis: reranker objective overweights semantic fluency/contextual breadth vs exact numeric-evidence grounding.
+
+2. Doc-level metrics saturate and hide problems.
+- Evidence: doc MRR fixed at 1.0 while chunk metrics degrade.
+- Hypothesis: relevant document is often retrieved, but best evidence chunk inside that document is not prioritized.
+
+3. Weak labels are not precision-usable.
+- Evidence: 121 FP out of 140 weak-positive rows in calibrated subset.
+- Hypothesis: doc-match score 0.7 is too permissive for relevance labeling in factual settings.
+
+4. NLI flags substantial unsupported/contradicted claim mass.
+- Evidence: contradiction ~0.40 and unsupported ~0.46-0.50.
+- Hypothesis: long answers contain extrapolative claims beyond retrieved evidence granularity.
+
+## 7) Actionable Next Steps
+
+1. Rerank objective/feature tuning with factual-priority constraints.
+- Add hard/soft boosts for period-aligned numeric/table chunks in rerank scoring.
+- Re-run E1 and require non-negative delta on chunk MRR and P@5 before promotion.
+
+2. Improve weak-label scheme.
+- Replace binary doc-match surrogate with graded weak labels including period/type alignment.
+- Keep manual 300+ audits for calibration and CIs.
+
+3. Expand manual audit slices where signal is weakest.
+- Increase factual sample beyond 140 rows and stratify by rerank disagreement bands.
+
+4. Keep this retrieval benchmark as a standing gate.
+- Run E1+E5+E6 for major retrieval/prompt changes and block merges on consistent factual rerank regressions.
+
+## 8) Repro Commands
+
+Scripts executed:
+- `agent_logs/scripts/eval/20260218_211000_run_retrieval_pool_and_metrics.sh`
+- `agent_logs/scripts/eval/20260218_215100_eval_retrieval_multi60.sh`
+- `agent_logs/scripts/eval/20260218_215700_summarize_retrieval_manual_sample.sh`
+
+Calibration command:
+- `source .venv/bin/activate && python scripts/calibrate_eval_metrics.py --labels-csv eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.codex_manual.csv --human-col human_relevance --weak-col weak_relevance --weak-threshold 0.5 --n-bootstrap 2000 --out-json eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.calibration.json`