Mlin/address benchmark findings

.env.example

@@ -48,20 +48,20 @@ ANN_HNSW_M=24
 ANN_HNSW_EF_CONSTRUCTION=200
 # Sparse retrieval method: bm25 (default, requires pg_textsearch on PG17/18) or fts.
 POSTGRES_SPARSE_SEARCH_METHOD=bm25
-# Narrative retrieval controls.
+# DEPRECATED FOR NOW: Narrative retrieval controls.
 # Query expansion is disabled by default to avoid user-query drift.
-FINRAG_ENABLE_NARRATIVE_QUERY_EXPANSION=0
+# FINRAG_ENABLE_NARRATIVE_QUERY_EXPANSION=0
 # Keep narrative aspect coverage enabled as a low-overhead faithfulness guardrail.
-FINRAG_ENABLE_NARRATIVE_ASPECT_COVERAGE=1
+# FINRAG_ENABLE_NARRATIVE_ASPECT_COVERAGE=1
 # RECREATE_ANN_INDEX=true
 # RESET_CORPUS=true
 # ALLOW_DEFAULT_SCHEMA_MUTATIONS=true
 DEBUG_SAMPLE_RATE=0.02
 DEBUG_MAX_SAMPLES=100
 DEBUG_SAMPLE_SEED=42
 
-# Path used by /ingested_companies endpoint.
-FINRAG_DOC_INDEX_PATH=./data/ingest_profiles/eval_revamp_combined_512_20260217/sec_filings_md_secparser/chunked_512_64/doc_index.jsonl
+# /ingested_companies resolves doc index from FINRAG_INGEST_PROFILE by default.
+# Keep FINRAG_DOC_INDEX_PATH unset in .env to avoid stale path drift across runs.
 
 # Optional LangSmith tracing (provider calls only, not app-level tracing)
 LANGSMITH_TRACING=false

AGENTS.md

@@ -10,6 +10,9 @@ Violations will cause runs to be blocked or reverted and large multimillion doll
 * Strictly follow existing style in the codebase.
 * Ensure every function is properly documented following the existing style.
 * Write in-line comments strategically, especially for key business logic. Do this sparingly, and strategically.
+* DEVELOP A STRONG DISTASTE FOR REGEX/TEXT HEURISTICS.
+    - Favor using established external libraries instead
+    - Favor using proper, generalized functions
 * Keep code concise, tasteful and elegant.
 * Don't overcomplicate things. Don't implement more than you need to.
 * Ensure code is readable by humans, easy to extend and maintain.
@@ -33,6 +36,7 @@ key/attribute existence, such as by using `dataclass`, `TypedDict` and class att
     * Do not run the linter after every change. It is too slow.
 * First, activate the python venv by running `source .venv/bin/activate` from the repository root.
 * Then, run `pre-commit run --all`.
+* IMPORTANT NOTE: due to sandbox permission errors, you will need to set `PRE_COMMIT_HOME` env var to `PRE_COMMIT_HOME=/tmp/pre-commit-cache` when running `pre-commit`.
 * We use the pyright pre-commit hook to catch typing issues. There may be a large number of such errors. Try your best to fix them where possible, and document your findings in the `agent_logs/LOGBOOK.md`. If fixing a particular error is too tedious, make a judgement as to whether you should just ignore it in-line, or modify the pyright config (if applicable).
 
 
@@ -82,13 +86,18 @@ you must ensure those comments continue to exist in the new/migrated function/co
     including modifications to `LOGBOOK.md`, `agent_logs/`, `CHANGELOG.md` and so on.
     - NEVER undo others' work.
     - Keep calm and continue executing with your plan. You do not need to stop and ask me about it.
+* WHEN WRITING UNIT TESTS:
+    - Monkeypatching is fine for isolating specific logic to test for, but you should NOT abuse it.
+    - You should still have proper "integration" style tests that pass in legitimate production-style inputs, and assert for legitimate outputs (as far as pragmatically possible).
+    - When dealing with tests that require external APIs like LLM APIs, of course monkeypatching helps to run these tests locally without actually calling the external LLM API and incurring costs. However, we should still set aside a comprehensive suite of integration tests that actually make the required LLM calls, and do comprehensive asserts/checks on the outputs. These tests can be run in special environment with LLM key available, and can be disabled by default for normal CI tests.
+    - But these tests should still exist.
 
 ## Testing rules
 
 * You must run the tests after wrapping up all changes, or before running an actual piece of work
 which relies on recent changes to the codebase, to ensure that core functions work as expected.
     - You don't need to run the tests after every little change. Exercise judgement.
 * First, activate the venv by running `source .venv/bin/activate` from the repository root.
-* Then, run tests with `pytest -vvv tests/`.
+* Then, run tests with `pytest tests/`.
 * Fix failing tests before proceeding.
 * Never bypass tests without explicit instruction.

BENCHMARK_FLAWED_PLANNER.md

@@ -0,0 +1,62 @@
+# Flawed Planner Prompt Regression (20 Feb)
+
+## Why this write-up exists
+I stopped the active run on request because the planner prompt appears too loose, causing excessive `clarification_required` decisions and downstream refusal behavior.
+
+## Run status at stop time
+- Completed:
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.baseline_best.single100.normal.tools12.norefine.20260220_001448`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.baseline_best.multi60.normal.tools12.norefine.20260220_002746`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.baseline_best.open200.normal.tools12.norefine.20260220_003443`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.ablation_no_rerank.single100.normal.tools12.norefine.20260220_010156`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.ablation_no_rerank.multi60.normal.tools12.norefine.20260220_011025`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.ablation_no_rerank.open200.normal.tools12.norefine.20260220_011749`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.ablation_no_material_cap.single100.normal.tools12.norefine.20260220_013728`
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.ablation_no_material_cap.multi60.normal.tools12.norefine.20260220_014954`
+- Interrupted mid-run:
+  - `eval/results_revamp/full_suite_ablation/eval_run.full_suite_ablation_20260220_001447.ablation_no_material_cap.open200.normal.tools12.norefine.20260220_020229`
+  - Partial artifact only (`generations.jsonl` with 5 rows), no scored summary.
+
+## Key regression signal
+- Current baseline `open200`:
+  - `faithfulness_v1` fail: `0.2764`
+  - `helpfulness_v1` fail: `0.4372`
+- Previous reduced-heuristics reference (`2026-02-18`):
+  - `eval/results_revamp/full_suite/eval_run.reduced_heuristics_full_retry4_envoverride_20260218_195034.open200.normal.tools12.norefine.20260218_202301`
+  - `faithfulness_v1` fail: `0.1350`
+  - `helpfulness_v1` fail: `0.0050`
+
+## Root-cause diagnosis
+The biggest change is planner behavior, not reranker toggles:
+
+- Previous run planner actions (`open200`, n=200):
+  - `answer`: `199`
+  - `clarification_required`: `1`
+- Current baseline planner actions (`open200`, n=199 ok rows):
+  - `answer`: `167`
+  - `clarification_required`: `32`
+
+Every one of these `32` clarification cases triggered `refuse_unindexed_ticker_candidates` using bogus inferred symbols.
+
+Examples from traces:
+- ATI thesis query -> inferred candidates: `GC=F, ALI=F, RB=F, HO=F, PL=F`
+- GEV capital allocation query -> inferred candidates: `CAPEX, EDD`
+- IESC risk query -> inferred candidate: `TDOG`
+
+This creates false refusals even when the true ticker is indexed.
+
+## Quantified impact of this failure mode (current baseline open200)
+- `clarification_required + refuse_unindexed` cases: `32`
+- Helpfulness fails among those: `32/32`
+- Faithfulness fails among those: `28/32`
+- Share of all helpfulness fails explained by this single mode: `32/87` (~36.8%)
+
+## Additional ablation note
+- Turning off reranker improved open-ended faithfulness (`0.2764 -> 0.1950`) but did not fix helpfulness (`0.4372 -> 0.4300`), which is consistent with the planner/refusal issue dominating helpfulness failures.
+- Removing material-point cap severely worsened latency/tail behavior and did not provide clear quality upside in completed slices.
+
+## Immediate fix direction
+1. Planner prompt: tighten `clarification_required` criteria to avoid triggering when a valid indexed ticker is present and intent is answerable.
+2. Runtime guardrail: do not call `refuse_unindexed_ticker_candidates` when planner returns valid structured output with empty tickers and `clarification_required`.
+3. Ticker inference fallback: never treat generic finance tokens (`CAPEX`, `M&A`, etc.) as ticker candidates for refusal logic.
+4. Add regression tests with the exact failing queries above.

BENCHMARK_PLANNER.md

@@ -0,0 +1,166 @@
+# Benchmark: Planner Characteristics Evaluation
+
+_Last updated: 2026-02-19_
+
+## 1) Scope
+This report benchmarks planner-side multi-label classification quality for query characteristics.
+
+Goal:
+- verify whether planner output correctly identifies all applicable query characteristics before answer generation.
+
+Characteristics evaluated:
+- `comparison`
+- `market_data`
+- `financial_metrics`
+- `filing_narrative`
+- `period_scoped`
+- `simple_numeric`
+
+Dataset:
+- `eval/eval_queries_planner_characteristics_manual100_20260219.jsonl`
+- 100 manually curated queries (not LLM-generated), with explicit labels and rationale per row.
+
+## 2) Experiments Run
+
+| ID | Experiment | Command/script | Output artifacts | Purpose |
+|---|---|---|---|---|
+| P1 | Planner generation run | `agent_logs/scripts/eval/20260219_205324_run_planner_eval_suite_live.sh` | `planner_predictions.jsonl`, `planner_prediction_summary.json` | Run planner on all 100 queries with timeout/retry controls |
+| P2 | Planner scoring | `scripts/score_planner_eval.py` (invoked by P1) | `planner_scores.jsonl`, `planner_score_summary.json`, `planner_review.csv`, `planner_score_summary.md` | Compute exact/subset/precision/recall metrics and per-characteristic confusion |
+| P3 | Failure-pattern analysis | `agent_logs/scripts/eval/20260219_205431_analyze_planner_eval_run.sh` | `agent_logs/reports/planner_eval_20260219/planner_eval_analysis_20260219_205341.json` | Aggregate mismatch patterns, tag-level slices, action errors |
+
+Run directory:
+- `eval/results_planner/planner_eval_run.planner_live_manual100_20260219_205341.20260219_205341`
+
+## 3) Run Configuration
+
+- workers: `12`
+- planner timeout: `350s`
+- planner retries: `1`
+- queries run: `100`
+- generation errors: `0`
+
+Runtime summary:
+- avg query latency: `2347.22 ms`
+- wall time: `21404.30 ms`
+- throughput: `4.67 queries/s`
+
+## 4) Topline Results
+
+### 4.1 Overall
+
+| Metric | Value |
+|---|---:|
+| characteristic exact match rate | 0.6400 |
+| expected-subset recall rate | 0.7600 |
+| macro precision | 0.9117 |
+| macro recall | 0.9033 |
+| macro F1 | 0.8907 |
+| micro precision | 0.9101 |
+| micro recall | 0.8571 |
+| micro F1 | 0.8828 |
+| action accuracy (6 labeled action rows) | 0.6667 |
+
+Interpretation:
+- planner is generally precise and high-recall on most characteristics, but exact-match quality is limited by a concentrated error mode.
+
+### 4.2 Per-Characteristic Breakdown
+
+| Characteristic | Support | Precision | Recall | F1 | TP | FP | FN |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| comparison | 22 | 0.9091 | 0.9091 | 0.9091 | 20 | 2 | 2 |
+| market_data | 37 | 1.0000 | 0.9189 | 0.9577 | 34 | 0 | 3 |
+| financial_metrics | 28 | 0.9032 | 1.0000 | 0.9492 | 28 | 3 | 0 |
+| filing_narrative | 40 | 0.9500 | 0.9500 | 0.9500 | 38 | 2 | 2 |
+| period_scoped | 28 | 0.8485 | 1.0000 | 0.9180 | 28 | 5 | 0 |
+| simple_numeric | 34 | 0.7778 | 0.4118 | 0.5385 | 14 | 4 | 20 |
+
+Key point:
+- `simple_numeric` is the clear bottleneck (20 false negatives out of 34 support).
+
+## 5) Failure Analysis
+
+### 5.1 Dominant mismatch pattern
+Top mismatch:
+- `missing=simple_numeric | extra=-` occurred `20` times.
+
+These misses are concentrated in queries labeled:
+- `financial_metrics period_scoped simple_numeric` (20 rows)
+
+Observed pattern:
+- predicted set was consistently `financial_metrics period_scoped`, omitting `simple_numeric`.
+
+Examples:
+- `planner_eval_0023`: "What was AAPL's net income in 2025?"
+- `planner_eval_0024`: "What was MSFT's total revenue in FY2024?"
+- `planner_eval_0025`: "What was NVDA's gross margin in Q2 2025?"
+
+### 5.2 Tag-level quality
+
+| Tag group | n | exact_match_rate | subset_recall_rate |
+|---|---:|---:|---:|
+| `comparison filing_narrative` | 14 | 1.0000 | 1.0000 |
+| `filing_narrative market_data` | 8 | 1.0000 | 1.0000 |
+| `financial_metrics period_scoped analysis` | 8 | 1.0000 | 1.0000 |
+| `market_data simple_numeric` | 14 | 0.8571 | 0.8571 |
+| `market_data contextual` | 8 | 0.1250 | 1.0000 |
+| `comparison market_data` | 6 | 0.5000 | 1.0000 |
+| `financial_metrics period_scoped simple_numeric` | 20 | 0.0000 | 0.0000 |
+| `clarification ambiguous_ticker` | 2 | 0.0000 | 0.0000 |
+
+Interpretation:
+- the planner usually includes core characteristics, but often adds/removes secondary tags in contextual market queries.
+- the period-scoped numeric bucket is currently overfit to "financial metric trend" interpretation and misses direct numeric intent.
+
+### 5.3 Action errors
+
+Action-labeled rows: 6 (`4` refusal + `2` clarification-required)
+
+Action mismatches (2):
+- `planner_eval_0099`: expected `clarification_required`, predicted `refused`
+- `planner_eval_0100`: expected `clarification_required`, predicted `refused`
+
+Both are ambiguous watchlist/bank comparison prompts without explicit ticker names.
+
+## 6) Surprising Observations
+
+1. Planner throughput was high despite local vLLM setup.
+- 100 planner calls completed in ~21.4s wall-clock with 12 threads and no failures.
+
+2. `simple_numeric` under-classification is highly concentrated, not diffuse.
+- 20 misses are effectively one repeated decision pattern, not random noise.
+
+3. Prompt-level contradiction likely explains the biggest gap.
+- In planner few-shot examples (`src/andromeda/query/runtime.py`), a direct numeric period question ("What was AAPL net income in 2025?") is labeled as `[financial_metrics, period_scoped]` without `simple_numeric`.
+- That pattern mirrors the 20-row failure bucket almost exactly.
+
+4. Action metric is unstable due low support.
+- `action_accuracy=0.6667` is based on only 6 rows; this should not be over-interpreted.
+
+## 7) Practical Recommendations
+
+1. Fix planner few-shot labels before changing architecture.
+- Align examples so direct period-scoped single-value metric queries include `simple_numeric`.
+
+2. Expand action-labeled benchmark rows.
+- Increase clarification/refusal-labeled rows from 6 to at least 30 to reduce metric variance.
+
+3. Keep this planner eval as a standing gate.
+- Require non-regression on:
+  - `simple_numeric recall`
+  - overall `exact_match_rate`
+  - `expected_subset_recall_rate`
+
+## 8) Repro
+
+Run:
+```bash
+source .venv/bin/activate
+agent_logs/scripts/eval/20260219_205324_run_planner_eval_suite_live.sh
+```
+
+Analyze:
+```bash
+source .venv/bin/activate
+agent_logs/scripts/eval/20260219_205431_analyze_planner_eval_run.sh \
+  eval/results_planner/planner_eval_run.planner_live_manual100_20260219_205341.20260219_205341
+```