From f0cb71ea110e8cfb24336d789116bafe28a11ce2 Mon Sep 17 00:00:00 2001 From: Lin Min Htoo Date: Fri, 20 Feb 2026 21:37:00 +0800 Subject: [PATCH] chore: remove MD files for now --- BENCHMARK_REDUCED_HEURISTICS.md | 164 ----------------- BENCHMARK_RETRIEVAL.md | 178 ------------------- BENCHMARK_WITH_FIXED_PLANNER_20Feb.md | 245 -------------------------- 3 files changed, 587 deletions(-) delete mode 100644 BENCHMARK_REDUCED_HEURISTICS.md delete mode 100644 BENCHMARK_RETRIEVAL.md delete mode 100644 BENCHMARK_WITH_FIXED_PLANNER_20Feb.md diff --git a/BENCHMARK_REDUCED_HEURISTICS.md b/BENCHMARK_REDUCED_HEURISTICS.md deleted file mode 100644 index ec3e2e3..0000000 --- a/BENCHMARK_REDUCED_HEURISTICS.md +++ /dev/null @@ -1,164 +0,0 @@ -# Benchmark: Reduced-Heuristics Branch - -_Last updated: 2026-02-18_ - -## 1) Scope -This report covers Task 2 for branch `mlin/reduce-hardcoded-heuristics`: -- re-run the eval pipeline with current deploy-matched settings, -- analyze failures, -- perform a manual judge audit (Codex reasoning, no judge-LLM self-audit), -- compare against previously recorded benchmark baselines in `BENCHMARK.md`. - -## 2) Run Configuration -Core settings used: -- mode: `normal` -- tools: enabled -- refine: disabled -- generation workers: `12` (thread backend) -- query timeout/retries: `350s`, `1` -- judge workers: `12` -- judge context: `80000` -- judge timeout/retries: `350s`, `1` -- schema: `eval_revamp_combined_512_20260217` - -Run group manifest: -- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.manifest.json` - -Run dirs: -- `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.single100.normal.tools12.norefine.20260218_195034` -- `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.multi60.normal.tools12.norefine.20260218_200838` -- `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.open200.normal.tools12.norefine.20260218_202301` - -## 3) Topline Metrics (This Run) - -### 3.1 Generation throughput/latency -| suite | n | n_err | avg_total_ms | wall_total_ms | qps | -|---|---:|---:|---:|---:|---:| -| single100 | 100 | 1 | 59,540.45 | 846,854.45 | 0.1181 | -| multi60 | 60 | 0 | 134,343.77 | 708,687.42 | 0.0847 | -| open200 | 200 | 0 | 63,161.56 | 1,074,835.48 | 0.1861 | - -### 3.2 Judge-facing fail rates -- single100 (`score_summary.json`): - - factual fail: `0.0882` - - open faithfulness fail: `0.1333` - - refusal fail: `0.0000` - - distractor focus fail: `0.0667` -- multi60: - - comparison fail: `0.1167` -- open200: - - open faithfulness fail: `0.1350` - - open helpfulness fail: `0.0050` - -## 4) Comparison vs Previous Baseline (`BENCHMARK.md`) -Reference row in `BENCHMARK.md`: -- `baseline_normal`: factual fail `0.0857`, open faith fail `0.1667`, comparison fail `0.0167`. - -Comparison (closest axes): -- factual fail: `0.0882` (near parity; slightly worse by +0.0025) -- open faithfulness fail: `0.1333` (improved by -0.0334) -- comparison fail: `0.1167` (material regression, +0.1000) - -Interpretation: -- removing brittle heuristics did **not** materially hurt factual fail rate, -- faithfulness on open-ended remained improved versus historical baseline, -- comparison handling regressed strongly and is now the dominant quality gap on multi-ticker prompts. - -## 5) Timeout Incident Log (Generation) -Observed and captured in `agent_logs/LOGBOOK.md`: -- hard failure query: - - `query_id=1dd6251b-e62b-4e58-ae52-35a1253e14c3` - - question: "What was LITE's net income in its 10-Q filed 2026-02-04?" - - failure: timed out after 2 attempts (`350s` + retry), `n_err=1` - - scavenged output: no draft/final answer, no tool trace, generation error record persisted. -- recovered long-tail examples: - - `aada22de-6020-41aa-be15-5516f64b0aca` (MSFT total revenue) succeeded on retry. - - `598beb04-ec0c-4314-893e-2deb8f167179` (INTC vs NVDA comparison) succeeded on retry. - -### 5.1 Isolated replay of the LITE timeout query -To check whether this was purely batch-queue starvation, I ran the same query in isolation. - -- Query: `What was LITE's net income in its 10-Q filed 2026-02-04?` -- Probe A (`agent_logs/scripts/eval/20260218_220200_probe_lite_isolated_latency.sh`): - - direct single-call runtime probe with outer `timeout 500s` - - outcome: process timed out (`exit 124`) before returning -- Probe B (`agent_logs/scripts/eval/20260218_221400_probe_lite_single_eval_timeout350.sh`): - - single-query `run_eval` with `concurrency=1`, `query_timeout_s=350`, `query_max_retries=0` - - run dir: `agent_logs/reports/retrieval_eval_20260218/lite_single_eval_probe/eval_run.lite_isolated_timeout350.20260218_220015` - - outcome: success in `20601 ms` (`n_ok=1`, `n_err=0`), with tool trace and retrieved/reranked chunks present. - -Inference: -- The long-tail timeout is not only a batching artifact; isolated calls can still hit pathological slow behavior. -- However, the same query can also complete quickly in isolated eval mode, which is consistent with intermittent decode/runtime stalls rather than deterministic query complexity. - -## 6) Manual Judge Audit (Codex, Non-Circular) - -### 6.1 Method -To avoid circularity, the audit did **not** call the judge LLM. -- Built decision table: - - `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.judge_audit_manual/decision_audit.raw.csv` -- Full 698-row decision set was split into six shards and manually labeled by Codex workers using rubric-by-`judge_id` reasoning. -- Merged labeled output: - - `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.judge_audit_manual/decision_audit.codex_manual.csv` -- Reliability report: - - `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.judge_audit_manual/judge_reliability_report.codex_manual.json` - -### 6.2 Alignment summary (test split) -| judge | n_test | accuracy | precision_fail | recall_fail | Cohen's kappa (test) | notes | -|---|---:|---:|---:|---:|---:|---| -| faithfulness_v1 | 58 | 0.9828 | 0.8750 | 1.0000 | 0.9235 | strong alignment | -| factual_correctness_v1 | 9 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | tiny sample | -| helpfulness_v1 | 85 | 0.9882 | 1.0000 | 0.5000 | 0.6614 | under-calls fail cases | -| comparison_v1 | 15 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | aligned on this set | -| focus_v1 | 4 | 1.0000 | 0.0000 | 0.0000 | 1.0000 | no fail cases in test split | -| refusal_v1 | 5 | 0.8000 | 0.0000 | 0.0000 | 0.0000 | misses refusal-needed cases | - -### 6.3 Cohen's kappa on manually audited dev/test splits -Computed from `judge_reliability_report.codex_manual.json` using manual labels (`human_label`) vs judge decision (`judge_prediction`) per `judge_id`. - -| judge | kappa_dev | kappa_test | -|---|---:|---:| -| comparison_v1 | 1.0000 | 1.0000 | -| factual_correctness_v1 | 0.6479 | 1.0000 | -| faithfulness_v1 | 0.9233 | 0.9235 | -| focus_v1 | 1.0000 | 1.0000 | -| helpfulness_v1 | 0.2809 | 0.6614 | -| refusal_v1 | 0.0000 | 0.0000 | - -Aggregate agreement across all audited decisions: -- pooled kappa (dev): `0.8294` (`n=522`) -- pooled kappa (test): `0.8708` (`n=176`) -- macro-average kappa (dev/test): `0.6420` / `0.7641` -- sample-weighted kappa (dev/test): `0.5792` / `0.7828` - -Interpretation: -- Overall agreement is strong at pooled level. -- The weakest agreement remains in `helpfulness_v1` and `refusal_v1`, consistent with observed under-calling of fail cases. - -### 6.4 Key disagreement patterns -Confusion from full 698 labeled decisions: -- false positives: `4` total - - mostly faithfulness over-flags (`3`) and one factual false positive. -- false negatives: `10` total - - helpfulness under-flags (`6`) for non-responsive comparison/analysis answers, - - refusal under-flags (`3`) where out-of-scope prompts were met with clarification instead of refusal, - - faithfulness under-flag (`1`) on unsupported filing-availability claim. - -## 7) Genuine Pipeline Failures (from manual audit) -Main categories of true failures (`human_label=1`): -- comparison completeness failures (`comparison_v1`, `7`): model defers/clarifies instead of producing requested side-by-side analysis. -- open-ended faithfulness failures (`faithfulness_v1`, `29`): period mismatch and unsupported specific claims remain the largest category. -- refusal behavior gaps (`refusal_v1`, `3`): out-of-scope ticker prompts not refused strongly enough. -- helpfulness failures (`helpfulness_v1`, `8`): mostly non-answers/deferrals for requested comparative analysis. - -## 8) Surprising Findings and Hypotheses -1. Removing brittle heuristics improved maintainability without collapsing factual/open-ended quality. -2. Multi-ticker comparison degraded sharply; likely because previous heuristic scaffolding implicitly forced comparative structure. -3. Faithfulness judge quality is now relatively strong under Codex-manual audit; biggest remaining reliability issue is helpfulness/refusal under-calling. -4. Timeout outliers still materially affect wall-clock and can dominate throughput for small suites. - -## 9) Immediate Follow-ups -1. Improve comparison answer planning (explicit required-output structure for multi-ticker comparison prompts). -2. Tighten refusal policy for out-of-scope tickers (prefer explicit refusal over vague clarifying loops). -3. Add retry+continue safeguards for long-tail timed-out generations (already partially in place). -4. Keep judge audits separated from judge-model outputs (Codex-manual process retained as the non-circular check). diff --git a/BENCHMARK_RETRIEVAL.md b/BENCHMARK_RETRIEVAL.md deleted file mode 100644 index bc4a1aa..0000000 --- a/BENCHMARK_RETRIEVAL.md +++ /dev/null @@ -1,178 +0,0 @@ -# Benchmark: Retrieval + Rerank Quality (Reduced-Heuristics Branch) - -## 1) Scope -This report continues from the completed 300-sample manual retrieval audit and closes the retrieval-focused evaluation workstream. - -Primary goals: -- quantify retriever vs reranker behavior, -- quantify evidence-support quality on open-ended answers, -- audit retrieval relevance with non-circular manual labels (Codex reasoning, not judge-LLM), -- calibrate weak labels against manual labels. - -All runs below use the same reduced-heuristics full-suite manifest: -- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.manifest.json` - -## 2) Experiments Run - -| ID | Experiment | Input artifacts | Output artifacts | What it measures | -|---|---|---|---|---| -| E1 | Factual retrieval/rerank IR metrics (`single100`) | `eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.single100.normal.tools12.norefine.20260218_195034` | `retrieval_rerank_metrics.json`, `retrieval_rerank_metrics.csv`, `retrieval_nli_claim_support.csv` | Pre vs post rerank MRR/hit/precision/recall using factual gold evidence anchors | -| E2 | Open-ended NLI evidence support (`open200`) | `eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.open200.normal.tools12.norefine.20260218_202301` | `retrieval_rerank_metrics.json`, `retrieval_nli_claim_support.csv` | Claim support / contradiction / unsupported rates | -| E3 | Multi slice retrieval pass (`multi60`) | `eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.multi60.normal.tools12.norefine.20260218_200838` | `retrieval_rerank_metrics.json` | Completeness check for full-suite parity (no factual/open-ended rows in this slice) | -| E4 | Retrieval candidate pool build | all three run dirs above | `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_envoverride_20260218_195034.retrieval_pool.csv`, `.stats.json` | pooled chunk candidates for manual relevance auditing | -| E5 | Manual relevance audit (300 rows) | `...retrieval_pool.sample300.csv` | `...retrieval_pool.sample300.codex_manual.csv` | Human-proxy relevance labels via Codex reasoning (non-circular) | -| E6 | Weak-label calibration | `...sample300.codex_manual.csv` | `...sample300.calibration.json` | Weak-label precision/recall/alignment vs manual labels | -| E7 | Manual audit summary rollup | `...sample300.codex_manual.csv` | `agent_logs/reports/retrieval_eval_20260218/manual_sample300_summary.json`, `.md` | relevance prevalence, pre/post membership, rank movement, top-k relevance slices | - -## 3) Core Results - -### 3.1 Retriever vs reranker on factual anchors (E1) - -| Metric | Pre-rerank | Post-rerank | Delta | -|---|---:|---:|---:| -| factual_n | 34 | 34 | - | -| chunk MRR | 0.3092 | 0.1743 | -0.1349 | -| chunk win rate | - | - | 0.1765 | -| chunk precision@5 | 0.1118 | 0.0647 | -0.0471 | -| chunk precision@10 | 0.0647 | 0.0471 | -0.0176 | -| chunk precision@25 | 0.0294 | 0.0282 | -0.0012 | -| chunk recall@25 | 0.7353 | 0.7059 | -0.0294 | -| doc MRR | 1.0000 | 1.0000 | 0.0000 | - -Interpretation: -- On factual gold-anchor queries, current reranking is net negative on chunk-level relevance concentration. -- Doc-level MRR is saturated at 1.0 and is not discriminative for this run. - -### 3.2 NLI claim support on open-ended generations (E1/E2) - -| Slice | n_open_ended_scored | support_rate | contradiction_rate | unsupported_rate | -|---|---:|---:|---:|---:| -| `single100` subset | 30 | 0.1000 | 0.3958 | 0.5042 | -| `open200` | 120 | 0.1292 | 0.4115 | 0.4594 | - -Interpretation: -- Support is low and contradiction/unsupported are high. -- This aligns directionally with remaining faithfulness pressure points in answer-level evals. - -### 3.3 Multi slice parity check (E3) - -`multi60` has no factual or open-ended rows, so retrieval IR/NLI outputs are expectedly `NaN`/empty for these specific metric families. - -## 4) Manual 300-Sample Relevance Audit (E5/E7) - -Manual labels are from Codex reasoning on each row, not from the judge LLM. - -Source: -- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.codex_manual.csv` - -Summary: -- `n_labeled = 300` -- overall positive relevance rate = `0.4167` - -By query kind: - -| Kind | n | n_positive | positive_rate | -|---|---:|---:|---:| -| factual | 140 | 19 | 0.1357 | -| open_ended | 90 | 53 | 0.5889 | -| comparison | 55 | 44 | 0.8000 | -| distractor | 15 | 9 | 0.6000 | - -Pre/post membership buckets: - -| Bucket | n | n_positive | positive_rate | -|---|---:|---:|---:| -| both pre+post | 195 | 74 | 0.3795 | -| pre-only | 53 | 26 | 0.4906 | -| post-only | 52 | 25 | 0.4808 | - -Relevant-rank movement (rows present in both pre and post, relevance=1): -- n=74, promoted=32, demoted=37, unchanged=5, avg delta(post-pre)=+0.0676 - -By kind (same movement view): - -| Kind | n | promoted | demoted | unchanged | avg_delta(post-pre) | -|---|---:|---:|---:|---:|---:| -| factual | 15 | 4 | 9 | 2 | +2.8000 | -| open_ended | 38 | 19 | 17 | 2 | -1.8421 | -| comparison | 17 | 6 | 10 | 1 | +3.4118 | -| distractor | 4 | 3 | 1 | 0 | -6.2500 | - -Sample top-k relevance slices (not an unbiased absolute P@k estimator; useful as directional diagnostics): - -| Phase | k | n_rows_in_slice | n_positive | positive_rate | -|---|---:|---:|---:|---:| -| pre | 5 | 63 | 40 | 0.6349 | -| post | 5 | 64 | 37 | 0.5781 | -| pre | 10 | 128 | 60 | 0.4688 | -| post | 10 | 134 | 71 | 0.5299 | - -Interpretation: -- At very early ranks (top-5), audited relevance is lower post-rerank than pre-rerank. -- At top-10, post-rerank recovers and slightly exceeds pre-rerank in this sample. -- Factual and comparison rows show more demotions than promotions, consistent with E1 factual-anchor degradation. - -## 5) Weak-Label Calibration (E6) - -Source: -- `eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.calibration.json` - -Threshold: weak relevance >= 0.5 => relevant. - -| Metric | Value | -|---|---:| -| n | 140 | -| accuracy | 0.1357 | -| precision_1 | 0.1357 | -| recall_1 | 1.0000 | -| f1_1 | 0.2390 | -| balanced_accuracy | 0.5000 | -| tp / fp / tn / fn | 19 / 121 / 0 / 0 | - -Interpretation: -- Current weak labels are recall-maximal but extremely low precision (many false positives). -- They are suitable as high-recall candidate generation tags, not as ground-truth proxies for precision-sensitive decisions. - -## 6) Surprising Findings and Hypotheses - -1. Reranking currently hurts factual chunk concentration. -- Evidence: negative chunk MRR and precision deltas on factual anchors. -- Hypothesis: reranker objective overweights semantic fluency/contextual breadth vs exact numeric-evidence grounding. - -2. Doc-level metrics saturate and hide problems. -- Evidence: doc MRR fixed at 1.0 while chunk metrics degrade. -- Hypothesis: relevant document is often retrieved, but best evidence chunk inside that document is not prioritized. - -3. Weak labels are not precision-usable. -- Evidence: 121 FP out of 140 weak-positive rows in calibrated subset. -- Hypothesis: doc-match score 0.7 is too permissive for relevance labeling in factual settings. - -4. NLI flags substantial unsupported/contradicted claim mass. -- Evidence: contradiction ~0.40 and unsupported ~0.46-0.50. -- Hypothesis: long answers contain extrapolative claims beyond retrieved evidence granularity. - -## 7) Actionable Next Steps - -1. Rerank objective/feature tuning with factual-priority constraints. -- Add hard/soft boosts for period-aligned numeric/table chunks in rerank scoring. -- Re-run E1 and require non-negative delta on chunk MRR and P@5 before promotion. - -2. Improve weak-label scheme. -- Replace binary doc-match surrogate with graded weak labels including period/type alignment. -- Keep manual 300+ audits for calibration and CIs. - -3. Expand manual audit slices where signal is weakest. -- Increase factual sample beyond 140 rows and stratify by rerank disagreement bands. - -4. Keep this retrieval benchmark as a standing gate. -- Run E1+E5+E6 for major retrieval/prompt changes and block merges on consistent factual rerank regressions. - -## 8) Repro Commands - -Scripts executed: -- `agent_logs/scripts/eval/20260218_211000_run_retrieval_pool_and_metrics.sh` -- `agent_logs/scripts/eval/20260218_215100_eval_retrieval_multi60.sh` -- `agent_logs/scripts/eval/20260218_215700_summarize_retrieval_manual_sample.sh` - -Calibration command: -- `source .venv/bin/activate && python scripts/calibrate_eval_metrics.py --labels-csv eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.codex_manual.csv --human-col human_relevance --weak-col weak_relevance --weak-threshold 0.5 --n-bootstrap 2000 --out-json eval/results_revamp/full_suite/reduced_heuristics_full_retry4_retrieval_pool.sample300.calibration.json` diff --git a/BENCHMARK_WITH_FIXED_PLANNER_20Feb.md b/BENCHMARK_WITH_FIXED_PLANNER_20Feb.md deleted file mode 100644 index 295bfc4..0000000 --- a/BENCHMARK_WITH_FIXED_PLANNER_20Feb.md +++ /dev/null @@ -1,245 +0,0 @@ -# Benchmark With Fixed Planner (20 Feb 2026) - -## Goal -Re-run the requested 3-way end-to-end benchmark after the planner routing fix: -1. `baseline_best` -2. `ablation_no_rerank` -3. `ablation_no_material_cap` - -and report quality + latency behavior under the latest full-suite settings. - -## Setup -- Run group: `full_suite_ablation_20260220_022028` -- Driver script: `agent_logs/scripts/20260219_2358_run_full_suite_rerank_material_ablation.sh` -- Generation: - - `mode=normal` - - `concurrency=12` (thread backend) - - `query_timeout_s=350` - - `query_max_retries=1` - - deploy-matched retrieval settings (`top_k_retrieve=40`, `top_k_rerank=25` via normal preset) -- Judge: - - `judge_workers=12` - - `judge_context_chars=80000` - - `judge_timeout_s=350` - - `judge_max_retries=1` -- Query suites: - - `single100`: `eval/eval_queries_combined512_single_balanced100_validated_tol05_20260217.jsonl` - - `multi60`: `eval/eval_queries_combined512_multi_comparison60_validated_tol05_20260217.jsonl` - - `open200`: `eval/eval_queries_openended200_diverse_20260217_v1.jsonl` - -## Planner-Fix Impact Check (Before vs After) -Primary reference for "before" (flawed planner prompt/routing era): -- `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.baseline_best.open200.normal.tools12.norefine.20260220_003443` - -Current fixed-planner baseline: -- `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.open200.normal.tools12.norefine.20260220_024910` - -### Open200 (baseline_best) delta -- Faithfulness fail: `0.2764 -> 0.1200` (`-15.64` pp) -- Helpfulness fail: `0.4372 -> 0.2800` (`-15.72` pp) - -### Routing-trace evidence for root cause removal -`open200` planner/tool traces: -- Flawed run: - - planner actions: `answer=167`, `clarification_required=32` - - `refuse_unindexed_ticker_candidates=32` -- Fixed run: - - planner actions: `answer=200` - - `refuse_unindexed_ticker_candidates=0` - -Interpretation: the major helpfulness regression was primarily caused by erroneous clarification/refusal routing; the routing fix removed that failure mode. - -## Exact Experiments Run -### Baseline (`baseline_best`) -- single100: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.single100.normal.tools12.norefine.20260220_022029` -- multi60: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.multi60.normal.tools12.norefine.20260220_024054` -- open200: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.open200.normal.tools12.norefine.20260220_024910` - -### Reranker off (`ablation_no_rerank`) -- single100: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_rerank.single100.normal.tools12.norefine.20260220_031323` -- multi60: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_rerank.multi60.normal.tools12.norefine.20260220_032256` -- open200: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_rerank.open200.normal.tools12.norefine.20260220_033214` - -### Material cap off (`ablation_no_material_cap`) -- single100: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_material_cap.single100.normal.tools12.norefine.20260220_035608` -- multi60: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_material_cap.multi60.normal.tools12.norefine.20260220_041300` -- open200 (partial generation, completed scoring): `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_material_cap.open200.normal.tools12.norefine.20260220_042421` - -## Results -### single100 -| Experiment | Gen n_ok / n_err | Avg gen total ms | Factual fail | Factual helpfulness fail | Open faithfulness fail | Open helpfulness fail | Distractor focus fail | -|---|---:|---:|---:|---:|---:|---:|---:| -| baseline_best | 99 / 1 | 47,317 | 0.2857 | 0.2571 | 0.1034 | 0.2759 | 0.4000 | -| ablation_no_rerank | 100 / 0 | 40,139 | 0.2857 | 0.2571 | 0.0333 | 0.2667 | 0.4000 | -| ablation_no_material_cap | 99 / 1 | 57,097 | 0.3143 | 0.2571 | 0.1379 | 0.2759 | 0.3333 | - -### multi60 -| Experiment | Gen n_ok / n_err | Avg gen total ms | Comparison fail | Comparison helpfulness fail | -|---|---:|---:|---:|---:| -| baseline_best | 60 / 0 | 68,428 | 0.5167 | 0.5000 | -| ablation_no_rerank | 60 / 0 | 75,780 | 0.5000 | 0.5167 | -| ablation_no_material_cap | 60 / 0 | 99,446 | 0.5167 | 0.5167 | - -### open200 -| Experiment | Gen n_ok / n_err | Avg gen total ms | Open faithfulness fail | Open helpfulness fail | Notes | -|---|---:|---:|---:|---:|---| -| baseline_best | 200 / 0 | 56,696 | 0.1200 | 0.2800 | complete | -| ablation_no_rerank | 200 / 0 | 54,507 | 0.1500 | 0.2850 | complete | -| ablation_no_material_cap | 190 / 10* | 85,927** | 0.1684 | 0.2947 | partial generation; scored on 200 queries with 190 generated answers | - -\* `ablation_no_material_cap/open200`: 191 generation rows written, 1 hard timeout error row, 9 query IDs missing due stuck-tail termination. - -\** `ablation_no_material_cap/open200` avg ms computed from `timing_ms.total_ms` over the 190 successful generations (no `generation_summary.json` because run was interrupted). - -## Reliability/Failure Notes -### Timeouts and stuck-tail behavior -Observed generation hard timeouts: -- baseline single100: `b5c816f9-08d6-41ea-a5a4-1e06ce0acd4f` -- no-material-cap single100: `2dcc67c3-e597-485a-81e4-fbb8226880c0` -- no-material-cap open200: `4d51932a-0d08-4512-8cd1-9dae6d68f695` - -`no-material-cap/open200` entered a late stuck-tail state (high CPU, no output growth). The run was terminated and scored from produced outputs to avoid blocking indefinitely. - -Failed query (hard timeout after retry): -- `4d51932a-0d08-4512-8cd1-9dae6d68f695` -- Question: "Which operational bottlenecks or dependencies does APH (APH) explicitly acknowledge in 2026, and how could they impact future results? Cite sources." -- Captured output: no draft/final answer and empty tool trace on failure row. - -## Why Helpfulness Is Still High: Failure Inspection -A direct audit of baseline runs shows helpfulness failures are dominated by refusal-style outputs for out-of-index tickers, not primarily by weak synthesis on indexed names. - -### Failure decomposition (baseline_post_fix) -Using `review.csv` + `generations.jsonl` in the three baseline runs: -- `single100`: `21` helpfulness fails; `20/21` (`95.2%`) are refusal-style (`"I can't answer because these tickers are not indexed"`). -- `multi60`: `30` helpfulness fails; `30/30` (`100%`) are refusal-style. -- `open200`: `56` helpfulness fails; `56/56` (`100%`) are refusal-style. -- Combined: `107` helpfulness fails; `106/107` (`99.1%`) are refusal-style. - -Most frequent rejected tickers in helpfulness-fail rows: -- `MSFT` (`26`) -- `TSLA` (`24`) -- `META` (`21`) -- `AMZN` (`21`) -- `AAPL` (`19`) - -### Concrete examples -#### Example A: open-ended fail driven by out-of-index refusal -- run: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.open200.normal.tools12.norefine.20260220_024910` -- query_id: `2fb4b736-656f-422f-a9a7-7745c8a7ab37` -- question: "What were the main stated drivers of profitability changes for TSLA (TSLA) in 2026, and which of them look persistent versus temporary? Cite sources." -- answer: refusal (`TSLA` not indexed). -- judge outcome: helpfulness fail; rationale says question was not addressed and no analysis/citations were provided. - -#### Example B: comparison fail where one ticker is out-of-index -- run: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.multi60.normal.tools12.norefine.20260220_024054` -- query_id: `a57210b5-b1dc-4b54-bd6f-8b12712c6c46` -- question: "In 2025, how do AMZN (AMZN) and LITE (LITE) differ in strategy and competitive positioning?" -- answer: refusal (`AMZN` not indexed), no partial analysis for `LITE`. -- judge outcome: helpfulness fail; rationale highlights missing comparative analysis and missing sources. - -#### Example C: factual fail from out-of-index refusal -- run: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.single100.normal.tools12.norefine.20260220_022029` -- query_id: `cf2cf7a8-aa09-4aa9-9d0b-7fb911cfbf0b` -- question: "What was MSFT's net income in its 10-K filed 2025-07-30?" -- answer: refusal (`MSFT` not indexed). -- judge outcome: helpfulness fail; rationale says the request was not answered. - -#### Example D: genuine indexed-answer quality miss (non-refusal) -- run: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.baseline_best.single100.normal.tools12.norefine.20260220_022029` -- query_id: `cdcab831-39b6-4154-810a-279596cbe4d5` -- question: "What was GOOGL's net income in its 10-Q filed 2025-04-25?" -- answer: indexed ticker, but model claimed net income was not explicitly stated. -- judge outcome: helpfulness fail; rationale says the filing excerpt did include net income and the answer was wrong/verbose. - -### Conclusion from inspection -- The high helpfulness fail rate is mainly a **coverage mismatch** between eval queries and indexed ticker universe in this run, not solely a generation-quality collapse. -- Secondary issue (smaller): occasional extraction/reasoning misses on indexed questions (Example D). - -### Action implications -1. Expand index/query coverage alignment (ingest `MSFT`, `TSLA`, `META`, `AMZN`, `AAPL`) or split metrics into `in_index` vs `out_of_index` buckets. -2. For mixed comparison queries (one indexed, one not), return partial answer for indexed ticker plus explicit limitation note instead of hard refusal. -3. Keep separate tracking for true synthesis misses on indexed queries (like Example D), since these are the errors that retrieval/prompt improvements should target. - -## Interpretation -### 1) Fixed planner routing materially improved baseline quality -The large baseline open200 improvement versus the flawed-planner run strongly indicates the previous spike in helpfulness/faithfulness failures was mostly routing-induced, not a pure retrieval/generation quality collapse. - -### 2) Reranker-off is mixed, not a clear win -- `single100`: reranker-off improved open-ended faithfulness (`0.1034 -> 0.0333`) and latency. -- `open200`: reranker-off worsened both open-ended fail rates (`0.1200 -> 0.1500`, `0.2800 -> 0.2850`). -- `multi60`: slight tradeoff (`comparison fail` improves a bit; `comparison helpfulness` worsens a bit). - -Conclusion: current evidence does not support globally disabling reranker for e2e default behavior. - -### 3) Removing material-point cap is net negative -- Quality generally degrades (single and open). -- Latency worsens substantially (especially `multi60`, and partial `open200` shows much higher mean time and more retries). - -Conclusion: keep the material cap enabled. - -## Recommended Default (post-fix) -Based on this rerun set: -- Keep `baseline_best` as default. -- Keep reranker enabled. -- Keep material-point cap enabled. -- Keep timeout+retry policy (`350s`, `1` retry), but add better stuck-tail handling at runner level (future work: per-request watchdog + salvageable partial completion checkpoints). - -## Repro / Commands Used -Primary launcher: -- `bash agent_logs/scripts/20260219_2358_run_full_suite_rerank_material_ablation.sh` - -After interruption of `ablation_no_material_cap/open200`, scoring was completed manually: -- `python -m scripts.score_eval --run-dir eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_022028.ablation_no_material_cap.open200.normal.tools12.norefine.20260220_042421 --judge-workers 12 --judge-context-chars 80000 --judge-timeout-s 350 --judge-max-retries 1` - -# ROOT CAUSE OF TICKER MISMATCH: - -- Verified cause: `.env` contained a stale `FINRAG_DOC_INDEX_PATH` (`exp__chunk_1024_o128_tokenizer__ctx_none__index_m24_ef200`), and the benchmark launcher used: - - `DOC_INDEX_PATH="${FINRAG_DOC_INDEX_PATH:-}"` -- Because `FINRAG_DOC_INDEX_PATH` was already set, the launcher silently selected the wrong doc index, even though the eval query sets were built for `eval_revamp_combined_512_20260217`. -- Evidence: - - runtime command logs showed `--doc-index-path ./data/ingest_profiles/exp__chunk_1024_o128_tokenizer__ctx_none__index_m24_ef200/.../doc_index.jsonl`; - - `.env` had `FINRAG_DOC_INDEX_PATH=./data/ingest_profiles/exp__chunk_1024_o128_tokenizer__ctx_none__index_m24_ef200/sec_filings_md_secparser/doc_index.jsonl`; - - benchmark query files were `eval_queries_combined512_*_20260217.jsonl`, which are tied to the 512 ingest profile. -- Prevention implemented: - - eval launchers now resolve doc index from ingest profile by default and **ignore** stale `.env` `FINRAG_DOC_INDEX_PATH` unless an explicit override is provided via `DOC_INDEX_PATH` (or `FINRAG_DOC_INDEX_PATH_OVERRIDE`); - - `.env.example` no longer sets `FINRAG_DOC_INDEX_PATH` to avoid accidental drift. - -## Update: Fixed-Settings Baseline Rerun (Interrupted by time) - -This entry logs the latest rerun that used the fixed profile/path wiring (`full_suite_ablation_fixed_20260220_124150`). - -### Run artifacts -- single100: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_fixed_20260220_124150.baseline_best.single100.normal.tools12.norefine.20260220_124205` -- multi60: `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_fixed_20260220_124150.baseline_best.multi60.normal.tools12.norefine.20260220_125759` -- open200 (interrupted): `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_fixed_20260220_124150.baseline_best.open200.normal.tools12.norefine.20260220_131312` - -### Metrics captured before stop -| Slice | Status | Generation | Avg gen total ms | Key fail rates | -|---|---|---:|---:|---| -| single100 | complete + scored | 100 / 0 | 55,657 | factual `0.0857`, factual helpfulness `0.0286`, open faithfulness `0.1000`, open helpfulness `0.0000`, distractor focus `0.0667`, distractor helpfulness `0.0000` | -| multi60 | complete + scored | 60 / 0 | 141,083 | comparison `0.0000`, comparison helpfulness `0.0000` | -| open200 | interrupted (not scored) | 24 generated rows | n/a | run manually stopped before scoring | - -### Bootstrap 95% confidence intervals for captured fail rates -Bootstrap configuration: -- resamples: `20,000` -- seed: `42` -- source artifact: `agent_logs/reports/20260220_fixed_planner_baseline_bootstrap_ci.json` - -| Metric | n | fail rate | bootstrap 95% CI | -|---|---:|---:|---:| -| single100 factual fail | 35 | 0.0857 | [0.0000, 0.2000] | -| single100 factual helpfulness fail | 35 | 0.0286 | [0.0000, 0.0857] | -| single100 open faithfulness fail | 30 | 0.1000 | [0.0000, 0.2000] | -| single100 open helpfulness fail | 30 | 0.0000 | [0.0000, 0.0000] | -| single100 distractor focus fail | 15 | 0.0667 | [0.0000, 0.2000] | -| single100 distractor helpfulness fail | 15 | 0.0000 | [0.0000, 0.0000] | -| multi60 comparison fail | 60 | 0.0000 | [0.0000, 0.0000] | -| multi60 comparison helpfulness fail | 60 | 0.0000 | [0.0000, 0.0000] | - -Note: -- For all-zero empirical fail rates, nonparametric bootstrap returns `[0, 0]` because every resample remains all-zero. This reflects the observed sample; it does not imply true population uncertainty is exactly zero. - -### Notes -- Runner settings for completed slices: `concurrency=12`, `query_timeout_s=350`, `query_max_retries=1`, retry multiplier `1.25`, cap `600`. -- `open200` was intentionally stopped early due time constraints; no judge metrics are available for this partial run.