Skip to content

Mlin/address benchmark findings #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
linminhtoo merged 2 commits into from
Feb 20, 2026
Merged

Mlin/address benchmark findings #47

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a50fcea
chore
linminhtoo Feb 20, 2026
ffaa13a
doc: update README_EVAL
linminhtoo Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 12 additions & 42 deletions README_EVAL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,48 +42,7 @@ Rationale: in the controlled sweep (`agent_logs/reports/chunk_size_tradeoff_17Fe
- `judge_max_retries=1`
- active faithfulness rubric: materiality-calibrated `faithfulness_v1` in `src/andromeda/eval/judges.py`

## 2) Current Metrics Snapshot

### Full single-ticker suite (100 queries, all non-comparison kinds)
Run:
- `eval/results_revamp/single/eval_run.single_ext_chunk512_v1_normal_tools12_norefine_eval100.20260217_043752/score_summary.json`

Metrics:
- factual:
- `numeric_accuracy=0.60`
- `factual_correctness_v1` fail rate: `0.0571`
- factual `helpfulness_v1` fail rate: `0.0286`
- open-ended:
- `faithfulness_v1` fail rate: `0.2667`
- open-ended `helpfulness_v1` fail rate: `0.0`
- refusal:
- `refusal_v1` fail rate: `0.0`
- distractor:
- `focus_v1` fail rate: `0.0667`
- distractor `helpfulness_v1` fail rate: `0.0`

### Multi-ticker comparison snapshot
Run:
- `eval/results_revamp/multi/eval_run.multi_holistic_normal_v2_tools8_norefine_calibrated.20260216_234717/score_summary.json`

Metrics:
- comparison `comparison_v1` fail rate: `0.0417`
- comparison `helpfulness_v1` fail rate: `0.0`

### Open-ended stress + judge-tuning snapshot
Generation baseline run (200 diverse open-ended):
- `eval/results_revamp/open/eval_run.open_diverse200_iter0_baseline_normal_tools12_norefine_qt350_jt350.20260218_002122/score_summary.json`
- faithfulness fail: `0.215`
- helpfulness fail: `0.01`

Judge-iteration run (fixed generations, materiality-calibrated prompt):
- `eval/results_revamp/judge_tuning/eval_run.open200_judge_iter3_materiality.20260218_010749/score_summary.json`
- faithfulness fail: `0.08`
- helpfulness fail: `0.015`

Interpretation: the large faithfulness delta in judge-only rescoring indicates prior over-strict false positives; judge reliability should be interpreted with manual-audit calibration, not raw fail-rate alone.

## 3) One-Pass Full Eval Suite
## 2) One-Pass Full Eval Suite

### Quick path (assets already prepared)
```bash
Expand Down Expand Up @@ -216,3 +175,14 @@ Per run directory (`planner_eval_run.*`):
- macro/micro precision, recall, F1
- per-characteristic TP/FP/FN/TN with precision/recall/F1
- action accuracy for labeled action rows (`refused`, `clarification_required`)


## 3) Current Metrics Snapshot

### Full single-ticker suite (100 queries, all non-comparison kinds)
TO BE UPDATED

### Multi-ticker comparison snapshot
TO BE UPDATED

### Open-ended stress
Loading