From 37f8d5197ea106d7fc062398451d999ec892287d Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 14:48:06 +0200 Subject: [PATCH 1/8] docs: FinBERT fine-tuning design spec Grounds the proposal in the 2026-07-02 zero-signal analysis (sentiment uncorrelated with next-day move, model below the all-neutral baseline) and specs the only ceiling-changing option: fine-tune FinBERT on the move labels with a leak-free time-based split. Covers the COVID regime-shift problem in the natural 80/20 date cut, contract impact (split column in processed_data.csv), fallback behavior, and the sign-offs needed from Aurora and Nadi per AGENTS.md. --- .../2026-07-02-finbert-finetune-design.md | 152 ++++++++++++++++++ 1 file changed, 152 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-02-finbert-finetune-design.md diff --git a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md new file mode 100644 index 0000000..f5137d3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md @@ -0,0 +1,152 @@ +# Fine-tuning FinBERT on move labels — design + +**Date:** 2026-07-02 +**Owner:** Jack (proposal) — implementation lands in Nadi's classifier, so **requires Nadi's sign-off**; the split change touches Aurora's output, so **requires Aurora's sign-off** too. +**Status:** proposed — awaiting review + +## Problem + +The retune loop cannot improve accuracy because there is nothing to tune. Analysis of +the 2026-07-02 run (13,527 rows, 30 tickers, 2012-2020): + +| Measurement | Value | +|---|---| +| corr(prob_up − prob_down, pct_change) | **−0.008** (zero) | +| Model accuracy | 0.368 | +| Statistical-independence baseline (same marginals, random pairing) | 0.356 | +| All-neutral majority baseline | **0.467** | +| Accuracy on high-confidence rows (conf ≥ 0.9, n=5,307) | 0.352 (worse than average) | + +FinBERT's pretrained sentiment head is statistically independent of next-day price +direction on this dataset. The confusion matrix has no diagonal structure — every true +class receives the same ~25/47/28 prediction split. Threshold/boost/max_length tuning +reshuffles marginals of noise; the loop's threshold-lowering direction actually moves +predictions *away* from the majority class and below the 0.467 all-neutral baseline. + +The only lever that can change the ceiling is replacing the frozen sentiment head with +a head trained on our actual labels. + +## Goals + +1. **Measure the real ceiling:** fine-tune FinBERT end-to-end on `up`/`down`/`neutral` + move labels and report honest held-out accuracy. +2. **Leak-free evaluation:** time-based train/test split so no future information + reaches training. +3. **Contract-compatible:** the fine-tuned classifier still emits `predictions_test.csv` + per Handoff 2 — Sabina, Jack, and Freddi are unaffected downstream. + +## Non-goals + +- Hitting the 0.60 target. Expected outcome is ~0.45–0.52; if headlines reliably + predicted next-day direction the strategy would be an arbitrage machine. Part of this + increment's value is evidence for **renegotiating the target** with the team. +- Feature engineering beyond text (momentum, volume) — future increment. +- Per-retune-iteration retraining. Training runs once; the retune loop keeps tuning + cheap inference-time params on top of the fine-tuned model. +- Touching the dlt pipeline. + +## Design + +### 1. Train/test split (Aurora's contract — needs sign-off) + +`processed_data.csv` gains a `split` column (`train`/`test`), assigned **by date**: +train = rows before the cut, test = rows on/after it. + +**COVID caveat (measured):** the natural 80/20 cut lands at **2020-01-17**, making the +test set the COVID crash — label distribution shifts from 47/27/26 (neutral/up/down) +overall to 28/35/38. Options: + +- **(a) cut at 2019-01-01** — test = calendar 2019 + Jan-Jun 2020, dilutes but keeps + some regime shift; +- **(b) cut at 2020-01-17 (80/20)** and report the shift explicitly; +- **(c) end the dataset at 2019-12-31** and cut inside it — cleanest science, drops data. + +**Recommendation: (c)** for the headline result, with (b) reported as a secondary +"regime shift" number. Decision belongs to the team. + +Downstream contract impact: Nadi trains on `split=train` rows only and predicts +`split=test` rows only, so `predictions_test.csv` stays all-test (Sabina's existing +validation already enforces this — no change on her side). + +### 2. Training script (Nadi's module — needs sign-off) + +New `agents/finetune_finbert.py` (standalone, run manually, not part of the loop): + +- HF `Trainer` on `ProsusAI/finbert` with a fresh 3-label head + (`num_labels=3`, `ignore_mismatched_sizes=True`). +- Tokenize `article_title`, `max_length=128`; class weights from train distribution + (neutral-heavy) via weighted cross-entropy. +- 3 epochs, batch 16, lr 2e-5, MPS/CPU autodetect. ~13k rows → roughly 20-60 min on + an M-series Mac. Early stopping on a small validation slice (last 10% of *train* + dates, so validation is also time-ordered). +- Saves to `outputs/finbert_finetuned/` (gitignored — weights are ~440 MB). +- Emits `outputs/finetune_report.json`: train/val/test accuracy, per-class metrics, + train date range, seed, epochs. Reproducibility record. + +### 3. Inference integration (Nadi's template) + +`CLASSIFIER_TEMPLATE` gains one line: `MODEL_DIR = {model_dir}` — when +`outputs/finbert_finetuned/` exists, load it instead of `ProsusAI/finbert`; otherwise +fall back to the pretrained model (current behavior, so nothing breaks before the +first training run). The label mapping comes from the fine-tuned config (`up`/`down`/ +`neutral` directly — no more sentiment→direction translation for the fine-tuned path). + +The retune loop keeps working unchanged on top: threshold/boost/max_length still apply +to the fine-tuned model's softmax. + +### 4. Success criteria + +The increment is done when `finetune_report.json` exists with a leak-free test +accuracy, whatever the number. Decision rules for the team afterwards: + +- test accuracy ≥ 0.52 → adopt fine-tuned model as the loop's default; +- 0.467–0.52 → adopt, but reframe the target around the measured ceiling; +- < 0.467 (all-neutral baseline) → do not adopt; the honest recommendation becomes + "aggregate headlines per ticker-day and/or add price features, or reframe the task." + +## Data flow + +Unchanged shape: `processed_data.csv` → `predictions_test.csv` → `evaluation_report.json` +→ loop. New artifacts (`finbert_finetuned/`, `finetune_report.json`) sit outside the +handoff chain; only Nadi's generated script reads the model dir. + +## Error handling + +- Missing `split` column → training script fails loudly with a pointer to the Aurora + change; generated classifier falls back to pretrained (current behavior). +- Missing/corrupt fine-tuned model dir → same fallback, logged. +- No new failure modes in the loop itself. + +## Testing + +- Unit: split logic is date-monotonic (max train date < min test date). +- Unit: generated classifier prefers the fine-tuned dir when present, falls back when + absent (template string assertions, no GPU needed). +- Integration (slow, opt-in marker): 50-row smoke train run completes and emits a + valid `finetune_report.json`. +- Regression: full existing suite (27 tests) unaffected when no fine-tuned model exists. + +## Complexity & size + +| Item | Complexity | LOC | +|---|---|---| +| Aurora split column + date cut | low | ~15 | +| `agents/finetune_finbert.py` | medium | ~120 | +| Template/`generate_code` model-dir support | low | ~15 | +| Tests | low-medium | ~60 | +| Contract doc + mock_data updates | trivial | ~10 | +| **Total** | **Medium** | **~220** | + +## Sign-off needed (per AGENTS.md golden rules 2 & 6) + +| Owner | Change | Why | +|---|---|---| +| Aurora | `split` column in `processed_data.csv` | her output format | +| Nadi | training script + template change | his module | +| Team | split-date choice (a/b/c above) + target renegotiation | affects the headline result | + +## Deferred + +- Headline aggregation per ticker-day (multiple headlines → one prediction). +- Non-text features (momentum, volume) — would need a different model head. +- Per-iteration retraining driven by the retune loop. From ef571715d619f652434842e3d8e7e2f2a4cd73bd Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:00:23 +0200 Subject: [PATCH 2/8] feat: FinBERT fine-tuning on move labels (implements 2026-07-02 spec) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements docs/superpowers/specs/2026-07-02-finbert-finetune-design.md — the only ceiling-changing option after the zero-signal finding (pretrained sentiment is uncorrelated with next-day move). - Aurora: `split` column in processed_data.csv — time-based 80/20 date cut (train = earliest dates) so no future info leaks into training; optional `dataset_end` state param (e.g. "2019-12-31") keeps the COVID regime out of the test window. Extracted as _assign_split(). - New agents/finetune_finbert.py: standalone script, plain torch loop (no new deps), weighted CE for the neutral-heavy distribution, time-ordered validation slice, keeps best-val weights, writes outputs/finbert_finetuned/ + finetune_report.json. - Nadi template: MODEL_DIR support — loads fine-tuned weights when outputs/finbert_finetuned/ exists (id2label already up/down/neutral), falls back to the pretrained sentiment head otherwise; predicts only split=test rows when the input has a split column. - Tests: split monotonicity, dataset_end, val ordering, template fallback/preference, test-rows-only predictions, opt-in `slow` smoke train (pytest -m slow). Contract doc + mock_data updated. Touches Aurora's and Nadi's modules per the spec's sign-off table — implemented ahead of sign-off at Jack's direction; owners should review before this merges. --- agents/aurora_processing.py | 23 ++++- agents/finetune_finbert.py | 182 +++++++++++++++++++++++++++++++++++ agents/nadi_classifier.py | 41 ++++++-- agents/state.py | 2 + docs/data_contracts.md | 3 +- mock_data/processed_data.csv | 26 ++--- pyproject.toml | 2 + tests/test_classifier.py | 42 ++++++++ tests/test_finetune_split.py | 75 +++++++++++++++ 9 files changed, 372 insertions(+), 24 deletions(-) create mode 100644 agents/finetune_finbert.py create mode 100644 tests/test_finetune_split.py diff --git a/agents/aurora_processing.py b/agents/aurora_processing.py index 1cb667d..6521986 100644 --- a/agents/aurora_processing.py +++ b/agents/aurora_processing.py @@ -75,6 +75,24 @@ def _assign_label(pct_change, threshold): # Processing node # --------------------------------------------------------------------------- +def _assign_split(df, dataset_end=None): + """Add a time-based train/test `split` column: earliest 80% of dates = train, + rest = test. Date-based (not random) so no future information leaks into + training. `dataset_end` (YYYY-MM-DD) optionally drops later rows first — + e.g. "2019-12-31" keeps the COVID regime out of the test window (see + docs/superpowers/specs/2026-07-02-finbert-finetune-design.md).""" + if dataset_end: + before = len(df) + df = df[pd.to_datetime(df["date"]) <= pd.to_datetime(dataset_end)].reset_index(drop=True) + print(f" Dataset ended at {dataset_end}: dropped {before - len(df):,} later rows") + split_cut = pd.to_datetime(df["date"]).quantile(0.8) + df["split"] = ["train" if d <= split_cut else "test" + for d in pd.to_datetime(df["date"])] + print(f" Split at {split_cut.date()}: " + f"{(df['split'] == 'train').sum():,} train / {(df['split'] == 'test').sum():,} test") + return df + + def processing_node(state: PipelineState) -> dict: """LangGraph node — runs the full processing pipeline and returns the output file path as `processed_data_path` in the state.""" @@ -145,10 +163,13 @@ def processing_node(state: PipelineState) -> dict: ].reset_index(drop=True) print(f" Outliers dropped: {before_outliers - len(df):,}. Final: {len(df):,} rows") + # 5b. Time-based train/test split (and optional dataset end date). + df = _assign_split(df, state.get("dataset_end")) + # 6. Export (Handoff 1 schema from data_contracts.md) output_cols = [ "article_id", "date", "ticker", "article_title", - "price_t", "price_t1", "pct_change", "label", + "price_t", "price_t1", "pct_change", "label", "split", ] df[output_cols].to_csv(out_path, index=False) print(f"[processing] Written to {out_path}") diff --git a/agents/finetune_finbert.py b/agents/finetune_finbert.py new file mode 100644 index 0000000..d53c6cb --- /dev/null +++ b/agents/finetune_finbert.py @@ -0,0 +1,182 @@ +"""Fine-tune FinBERT on the up/down/neutral move labels (standalone, run once). + +Not part of the pipeline loop. Trains on processed_data.csv's split=train rows +(the last 10% of train *dates* held out for validation), evaluates on split=test, +saves the best-validation model to outputs/finbert_finetuned/ and a +finetune_report.json reproducibility record next to it. The generated classifier +(agents/nadi_classifier.py) picks the saved model up automatically on its next run. + +Design + motivation: docs/superpowers/specs/2026-07-02-finbert-finetune-design.md. + + uv run agents/finetune_finbert.py --data data/processed_data.csv +""" + +import argparse +import json +import os +import random + +import pandas as pd +import torch +from torch.utils.data import DataLoader, Dataset +from transformers import AutoModelForSequenceClassification, AutoTokenizer + +MODEL = "ProsusAI/finbert" +LABELS = ["up", "down", "neutral"] +LABEL2ID = {name: i for i, name in enumerate(LABELS)} +HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN") + + +class HeadlineDataset(Dataset): + def __init__(self, titles, labels, tokenizer, max_length): + self.enc = tokenizer(list(titles), truncation=True, max_length=max_length, + padding="max_length", return_tensors="pt") + self.labels = torch.tensor([LABEL2ID[label] for label in labels]) + + def __len__(self): + return len(self.labels) + + def __getitem__(self, i): + return {"input_ids": self.enc["input_ids"][i], + "attention_mask": self.enc["attention_mask"][i], + "labels": self.labels[i]} + + +def _device(): + if torch.backends.mps.is_available(): + return torch.device("mps") + if torch.cuda.is_available(): + return torch.device("cuda") + return torch.device("cpu") + + +def _split_frames(df: pd.DataFrame): + """train/val/test frames. Validation is the last 10% of *train* dates so it + is time-ordered like the real test set (no random leakage).""" + if "split" not in df.columns: + raise SystemExit( + "processed_data.csv has no 'split' column — regenerate it with the " + "updated Processing agent (agents/aurora_processing.py) first." + ) + train_all = df[df["split"] == "train"] + test = df[df["split"] == "test"] + val_cut = pd.to_datetime(train_all["date"]).quantile(0.9) + is_val = pd.to_datetime(train_all["date"]) > val_cut + return train_all[~is_val], train_all[is_val], test + + +@torch.no_grad() +def evaluate(model, loader, device): + model.eval() + correct_total, per_class = 0, {i: [0, 0] for i in range(len(LABELS))} # hits, count + for batch in loader: + preds = model(input_ids=batch["input_ids"].to(device), + attention_mask=batch["attention_mask"].to(device)).logits.argmax(dim=1).cpu() + for pred, true in zip(preds, batch["labels"]): + per_class[int(true)][1] += 1 + if pred == true: + per_class[int(true)][0] += 1 + correct_total += 1 + n = sum(count for _, count in per_class.values()) + return (correct_total / n if n else 0.0, + {LABELS[i]: round(hits / count, 4) if count else 0.0 + for i, (hits, count) in per_class.items()}) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--data", default="data/processed_data.csv") + ap.add_argument("--out-dir", default="outputs/finbert_finetuned") + ap.add_argument("--epochs", type=int, default=3) + ap.add_argument("--batch-size", type=int, default=16) + ap.add_argument("--lr", type=float, default=2e-5) + ap.add_argument("--max-length", type=int, default=128) + ap.add_argument("--seed", type=int, default=42) + ap.add_argument("--limit", type=int, default=None, + help="cap rows per split (smoke tests)") + args = ap.parse_args() + + random.seed(args.seed) + torch.manual_seed(args.seed) + + df = pd.read_csv(args.data) + train, val, test = _split_frames(df) + if args.limit: + train, val, test = train.head(args.limit), val.head(args.limit), test.head(args.limit) + print(f"[finetune] rows: {len(train)} train / {len(val)} val / {len(test)} test " + f"| train dates {train['date'].min()}..{train['date'].max()}") + + device = _device() + tokenizer = AutoTokenizer.from_pretrained(MODEL, token=HF_TOKEN) + model = AutoModelForSequenceClassification.from_pretrained( + MODEL, token=HF_TOKEN, num_labels=len(LABELS), + id2label={i: name for i, name in enumerate(LABELS)}, label2id=LABEL2ID, + ignore_mismatched_sizes=True, # fresh 3-way head over the sentiment one + ).to(device) + + def loader(frame, shuffle=False): + return DataLoader( + HeadlineDataset(frame["article_title"], frame["label"], tokenizer, args.max_length), + batch_size=args.batch_size, shuffle=shuffle) + + train_loader, val_loader, test_loader = loader(train, shuffle=True), loader(val), loader(test) + + # Weighted CE so the neutral-heavy train distribution doesn't collapse the + # up/down classes to noise. + counts = train["label"].value_counts() + weights = torch.tensor([len(train) / (len(LABELS) * counts.get(name, 1)) + for name in LABELS], dtype=torch.float).to(device) + loss_fn = torch.nn.CrossEntropyLoss(weight=weights) + optimizer = torch.optim.AdamW(model.parameters(), lr=args.lr) + + best_val, history = -1.0, [] + os.makedirs(args.out_dir, exist_ok=True) + for epoch in range(1, args.epochs + 1): + model.train() + running = 0.0 + for step, batch in enumerate(train_loader, 1): + optimizer.zero_grad() + logits = model(input_ids=batch["input_ids"].to(device), + attention_mask=batch["attention_mask"].to(device)).logits + loss = loss_fn(logits, batch["labels"].to(device)) + loss.backward() + optimizer.step() + running += loss.item() + if step % 100 == 0: + print(f" epoch {epoch} step {step}/{len(train_loader)} " + f"loss {running / step:.4f}") + val_acc, _ = evaluate(model, val_loader, device) + history.append({"epoch": epoch, "train_loss": round(running / max(len(train_loader), 1), 4), + "val_accuracy": round(val_acc, 4)}) + print(f"[finetune] epoch {epoch}: val accuracy {val_acc:.4f}") + if val_acc > best_val: # keep the best-validation weights, not the last + best_val = val_acc + model.save_pretrained(args.out_dir) + tokenizer.save_pretrained(args.out_dir) + + # Score the SAVED (best-val) model on the held-out test split. + best = AutoModelForSequenceClassification.from_pretrained(args.out_dir).to(device) + test_acc, test_class = evaluate(best, test_loader, device) + print(f"[finetune] test accuracy {test_acc:.4f} per-class {test_class}") + + report = { + "base_model": MODEL, + "model_dir": args.out_dir, + "test_accuracy": round(test_acc, 4), + "test_class_accuracy": test_class, + "best_val_accuracy": round(best_val, 4), + "epochs": history, + "train_rows": len(train), "val_rows": len(val), "test_rows": len(test), + "train_date_range": [str(train["date"].min()), str(train["date"].max())], + "test_date_range": [str(test["date"].min()), str(test["date"].max())], + "params": {"epochs": args.epochs, "batch_size": args.batch_size, "lr": args.lr, + "max_length": args.max_length, "seed": args.seed, "limit": args.limit}, + } + report_path = os.path.join(os.path.dirname(args.out_dir) or ".", "finetune_report.json") + with open(report_path, "w", encoding="utf-8") as f: + json.dump(report, f, indent=2) + print(f"[finetune] report written to {report_path}") + + +if __name__ == "__main__": + main() diff --git a/agents/nadi_classifier.py b/agents/nadi_classifier.py index 5f460fe..9e93d59 100644 --- a/agents/nadi_classifier.py +++ b/agents/nadi_classifier.py @@ -32,6 +32,7 @@ from transformers import AutoTokenizer, AutoModelForSequenceClassification MODEL = "ProsusAI/finbert" +MODEL_DIR = {model_dir} # fine-tuned weights dir; None -> pretrained sentiment head MAX_LENGTH = {max_length} THRESHOLD = {threshold} FOCUS_LABELS = {focus_labels} @@ -42,12 +43,18 @@ # faster downloads); fall back to unauthenticated access when it is absent. HF_TOKEN = os.environ.get("HF_TOKEN") or os.environ.get("HUGGING_FACE_HUB_TOKEN") -tokenizer = AutoTokenizer.from_pretrained(MODEL, token=HF_TOKEN) -model = AutoModelForSequenceClassification.from_pretrained(MODEL, token=HF_TOKEN) +if MODEL_DIR and os.path.isdir(MODEL_DIR): + # Fine-tuned on our move labels: id2label is already up/down/neutral. + tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR) + model = AutoModelForSequenceClassification.from_pretrained(MODEL_DIR) + ID2OURS = {{i: model.config.id2label[i].lower() for i in model.config.id2label}} +else: + # Pretrained sentiment head: translate sentiment -> move direction. + tokenizer = AutoTokenizer.from_pretrained(MODEL, token=HF_TOKEN) + model = AutoModelForSequenceClassification.from_pretrained(MODEL, token=HF_TOKEN) + ID2OURS = {{i: SENTIMENT_TO_LABEL[model.config.id2label[i].lower()] for i in model.config.id2label}} model.eval() -ID2OURS = {{i: SENTIMENT_TO_LABEL[model.config.id2label[i].lower()] for i in model.config.id2label}} - def classify(title: str) -> dict: inputs = tokenizer(title, return_tensors="pt", truncation=True, max_length=MAX_LENGTH) with torch.no_grad(): @@ -90,11 +97,19 @@ def main(src: str, dst: str) -> None: if not rows: raise ValueError("Source file is empty") - out_cols = list(rows[0].keys()) + [ + # When Aurora supplies a train/test split, predict only the held-out test + # rows (training rows were seen by the fine-tuned model). Without a split + # column every row is treated as test (pre-split behavior). + if "split" in rows[0]: + rows = [r for r in rows if r["split"] == "test"] + if not rows: + raise ValueError("No split=test rows in source file") + + # split goes last per the contract, wherever it sat in the input. + out_cols = [c for c in rows[0].keys() if c != "split"] + [ "predicted_label", "confidence", "prob_up", "prob_down", "prob_neutral", "split" ] - - # Process all rows and mark them as test split + for row in rows: pred_data = classify(row["article_title"]) row.update(pred_data) @@ -131,11 +146,19 @@ def generate_code(state: PipelineState) -> dict: code_path = state.get("classifier_code_path") or os.path.join(OUTPUT_DIR, "classifier.py") os.makedirs(os.path.dirname(code_path) or ".", exist_ok=True) + # Prefer fine-tuned weights when a training run has produced them (see + # agents/finetune_finbert.py); otherwise the generated script falls back to + # the pretrained sentiment head. + model_dir = state.get("model_dir") or os.path.join(OUTPUT_DIR, "finbert_finetuned") + if not os.path.isdir(model_dir): + model_dir = None + formatted_code = CLASSIFIER_TEMPLATE.format( threshold=threshold, max_length=max_length, focus_labels=repr(focus_labels), - boost_factor=boost_factor + boost_factor=boost_factor, + model_dir=repr(model_dir) ) with open(code_path, "w", encoding="utf-8") as f: @@ -154,7 +177,7 @@ def generate_code(state: PipelineState) -> dict: f.write(formatted_code) metadata = { - "model_name": "ProsusAI/finbert", + "model_name": model_dir or "ProsusAI/finbert", "fine_tuning_params": { "threshold": threshold, "max_length": max_length, diff --git a/agents/state.py b/agents/state.py index 97fa6c3..e953548 100644 --- a/agents/state.py +++ b/agents/state.py @@ -12,6 +12,8 @@ class PipelineState(TypedDict, total=False): # ── Runtime config ─────────────────────────────────────────────────────── threshold: float # price-change boundary (default 0.01 = ±1%) data_dir: str | None # repo root; agents resolve file paths from here + dataset_end: str | None # optional YYYY-MM-DD; Aurora drops later rows (e.g. + # "2019-12-31" keeps COVID out of the test window) loop_iteration: int # retune loop counter, starts at 1 # ── Handoff 1: Processing → Classifier ────────────────────────────────── diff --git a/docs/data_contracts.md b/docs/data_contracts.md index b241fcc..6583f24 100644 --- a/docs/data_contracts.md +++ b/docs/data_contracts.md @@ -24,6 +24,7 @@ Built by joining FNSPID headlines to yfinance prices on `ticker` + publication d | price_t1 | float | 125.12 | closing price on next trading day (from yfinance) | | pct_change | float | 3.38 | percentage change T to T+1 | | label | string | up | >+1% = up, <-1% = down, in between = neutral | +| split | string | train | time-based: earliest 80% of dates = `train`, rest = `test` — so no future information leaks into fine-tuning. Nadi trains on `train` rows and predicts `test` rows only | ## Handoff 2 — Classifier Agent (Nadi) → Evaluator Agent (Sabina) @@ -56,7 +57,7 @@ Nadi receives `processed_data.csv` from Aurora and adds the prediction columns. | prob_up | float | 0.87 | **added by Nadi** — softmax probability for `up` | | prob_down | float | 0.05 | **added by Nadi** — softmax probability for `down` | | prob_neutral | float | 0.08 | **added by Nadi** — softmax probability for `neutral` (`prob_up + prob_down + prob_neutral ≈ 1`) | -| split | string | test | **added by Nadi** — all rows in this file must be test rows only | +| split | string | test | all rows in this file must be test rows only — Nadi predicts only Aurora's `split=test` rows (train rows were seen by the fine-tuned model; see `agents/finetune_finbert.py`) | ## Handoff 3 — Evaluator Agent (Sabina) → Manager Agent (Jack) diff --git a/mock_data/processed_data.csv b/mock_data/processed_data.csv index d2a89f0..183fc17 100644 --- a/mock_data/processed_data.csv +++ b/mock_data/processed_data.csv @@ -1,13 +1,13 @@ -article_id,date,ticker,article_title,price_t,price_t1,pct_change,label -FNSPID_00001,2021-03-15,AAPL,Apple unveils new iPhone lineup as analysts turn upbeat,121.03,125.12,3.38,up -FNSPID_00002,2021-04-02,MSFT,Microsoft cloud revenue beats expectations,242.35,248.00,2.33,up -FNSPID_00003,2021-05-10,TSLA,Tesla recalls vehicles over software glitch,672.37,629.04,-6.44,down -FNSPID_00004,2021-06-21,AMZN,Amazon faces antitrust scrutiny in Europe,167.10,165.00,-1.26,down -FNSPID_00005,2021-07-08,NVDA,Nvidia announces stock split as shares hold steady,195.00,195.80,0.41,neutral -FNSPID_00006,2021-08-12,GOOG,Google ad sales steady amid market caution,137.50,137.10,-0.29,neutral -FNSPID_00007,2021-09-15,META,Meta invests heavily in VR as investors stay wary,376.00,369.50,-1.73,down -FNSPID_00008,2021-10-20,NFLX,Netflix subscriber growth tops forecasts,615.00,640.20,4.10,up -FNSPID_00009,2021-11-03,AMD,AMD lands major data center contract,120.50,126.30,4.81,up -FNSPID_00010,2021-12-01,INTC,Intel delays next-gen chip launch,52.00,51.20,-1.54,down -FNSPID_00011,2022-01-25,AAPL,Apple supply chain concerns ease,160.00,161.10,0.69,neutral -FNSPID_00012,2022-02-14,MSFT,Microsoft announces Activision acquisition,295.00,300.50,1.86,up +article_id,date,ticker,article_title,price_t,price_t1,pct_change,label,split +FNSPID_00001,2021-03-15,AAPL,Apple unveils new iPhone lineup as analysts turn upbeat,121.03,125.12,3.38,up,train +FNSPID_00002,2021-04-02,MSFT,Microsoft cloud revenue beats expectations,242.35,248.0,2.33,up,train +FNSPID_00003,2021-05-10,TSLA,Tesla recalls vehicles over software glitch,672.37,629.04,-6.44,down,train +FNSPID_00004,2021-06-21,AMZN,Amazon faces antitrust scrutiny in Europe,167.1,165.0,-1.26,down,train +FNSPID_00005,2021-07-08,NVDA,Nvidia announces stock split as shares hold steady,195.0,195.8,0.41,neutral,train +FNSPID_00006,2021-08-12,GOOG,Google ad sales steady amid market caution,137.5,137.1,-0.29,neutral,train +FNSPID_00007,2021-09-15,META,Meta invests heavily in VR as investors stay wary,376.0,369.5,-1.73,down,train +FNSPID_00008,2021-10-20,NFLX,Netflix subscriber growth tops forecasts,615.0,640.2,4.1,up,train +FNSPID_00009,2021-11-03,AMD,AMD lands major data center contract,120.5,126.3,4.81,up,train +FNSPID_00010,2021-12-01,INTC,Intel delays next-gen chip launch,52.0,51.2,-1.54,down,test +FNSPID_00011,2022-01-25,AAPL,Apple supply chain concerns ease,160.0,161.1,0.69,neutral,test +FNSPID_00012,2022-02-14,MSFT,Microsoft announces Activision acquisition,295.0,300.5,1.86,up,test diff --git a/pyproject.toml b/pyproject.toml index 57697fe..5e708c6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -27,3 +27,5 @@ dev = [ [tool.pytest.ini_options] pythonpath = ["."] # import `agents` package from repo root +markers = ["slow: opt-in tests that download models / train (deselect by default)"] +addopts = "-m 'not slow'" diff --git a/tests/test_classifier.py b/tests/test_classifier.py index a341059..d9f299c 100644 --- a/tests/test_classifier.py +++ b/tests/test_classifier.py @@ -109,6 +109,48 @@ def test_classifier_archives_each_iteration_separately(outdir): assert os.path.exists(second["classifier_history_path"]) assert first["classifier_history_path"] != second["classifier_history_path"] +def test_generated_code_prefers_finetuned_model_when_present(outdir, monkeypatch): + """generate_code points the script at outputs/finbert_finetuned when the dir + exists, and falls back to the pretrained hub model (MODEL_DIR = None) when + it doesn't. Template-level check — no model download.""" + from agents.nadi_classifier import generate_code + + # No fine-tuned dir -> fallback + code_path = outdir / "classifier.py" + res = generate_code({"classifier_code_path": str(code_path)}) + content = code_path.read_text() + assert "MODEL_DIR = None" in content + assert res["classifier_metadata"]["model_name"] == "ProsusAI/finbert" + + # Fine-tuned dir present -> preferred + model_dir = outdir / "finbert_finetuned" + model_dir.mkdir() + res = generate_code({"classifier_code_path": str(code_path), + "model_dir": str(model_dir)}) + content = code_path.read_text() + assert f"MODEL_DIR = {str(model_dir)!r}" in content + assert res["classifier_metadata"]["model_name"] == str(model_dir) + + +def test_predictions_only_contain_test_split_rows(outdir): + """With Aurora's split column in the input, the generated classifier predicts + only split=test rows (train rows were seen in fine-tuning).""" + df = pd.read_csv(PROCESSED_DATA) + assert "split" in df.columns and (df["split"] == "train").any() + + code_path = outdir / "classifier.py" + pred_path = outdir / "predictions_test.csv" + ClassifierAgent().run(processed_data=PROCESSED_DATA, + classifier_code=str(code_path), predictions=str(pred_path)) + + out = pd.read_csv(pred_path) + assert len(out) == (df["split"] == "test").sum() + assert (out["split"] == "test").all() + assert set(out["article_id"]) == set(df[df["split"] == "test"]["article_id"]) + # split stays the last column per the contract + assert list(out.columns)[-1] == "split" + + def test_classifier_to_evaluator_integration(outdir): from agents.sabina_evaluator import EvaluatorAgent diff --git a/tests/test_finetune_split.py b/tests/test_finetune_split.py new file mode 100644 index 0000000..f9a202e --- /dev/null +++ b/tests/test_finetune_split.py @@ -0,0 +1,75 @@ +"""Tests for the fine-tune increment's plumbing: Aurora's time-based split and +the finetune script's frame splitting. Model training itself is exercised by the +opt-in smoke test at the bottom (deselected by default via the `slow` marker). +""" + +import pandas as pd +import pytest + +from agents.aurora_processing import _assign_split +from agents.finetune_finbert import _split_frames + + +def _frame(dates): + labels = ["up", "down", "neutral"] + return pd.DataFrame({ + "date": dates, + "article_title": [f"headline {i}" for i in range(len(dates))], + "label": [labels[i % 3] for i in range(len(dates))], + }) + + +def test_split_is_date_monotonic(): + dates = [f"2019-{m:02d}-15" for m in range(1, 11)] + df = _assign_split(_frame(dates)) + train_max = pd.to_datetime(df[df["split"] == "train"]["date"]).max() + test_min = pd.to_datetime(df[df["split"] == "test"]["date"]).min() + assert train_max < test_min + assert set(df["split"]) == {"train", "test"} + + +def test_dataset_end_drops_later_rows(): + dates = ["2019-06-01", "2019-12-30", "2020-03-01", "2020-06-01"] + df = _assign_split(_frame(dates), dataset_end="2019-12-31") + assert len(df) == 2 + assert pd.to_datetime(df["date"]).max() <= pd.Timestamp("2019-12-31") + + +def test_split_frames_validation_is_time_ordered(): + dates = [f"2019-{m:02d}-01" for m in range(1, 11)] + df = _assign_split(_frame(dates)) + train, val, test = _split_frames(df) + assert len(val) > 0 + # val comes after train in time, and both precede test + assert pd.to_datetime(train["date"]).max() < pd.to_datetime(val["date"]).min() + assert pd.to_datetime(val["date"]).max() <= pd.to_datetime(test["date"]).min() + + +def test_split_frames_requires_split_column(): + with pytest.raises(SystemExit, match="split"): + _split_frames(pd.DataFrame({"date": ["2019-01-01"], "label": ["up"]})) + + +@pytest.mark.slow +def test_finetune_smoke(tmp_path, monkeypatch): + """End-to-end 20-row training run. Slow (downloads FinBERT, trains 1 epoch): + run explicitly with `uv run pytest -m slow tests/test_finetune_split.py`.""" + import sys + from agents import finetune_finbert + + dates = ([f"2019-{m:02d}-{d:02d}" for m in range(1, 11) for d in (1, 15)]) + df = _assign_split(_frame(dates)) + data = tmp_path / "processed_data.csv" + df.to_csv(data, index=False) + out_dir = tmp_path / "finbert_finetuned" + + monkeypatch.setattr(sys, "argv", [ + "finetune_finbert.py", "--data", str(data), "--out-dir", str(out_dir), + "--epochs", "1", "--batch-size", "4", "--limit", "20", + ]) + finetune_finbert.main() + + import json + report = json.loads((tmp_path / "finetune_report.json").read_text()) + assert 0.0 <= report["test_accuracy"] <= 1.0 + assert (out_dir / "config.json").exists() From 6d876c4a7ba19de103dff9062806336ab7c9ded0 Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:17:56 +0200 Subject: [PATCH 3/8] =?UTF-8?q?docs:=20fine-tune=20results=20=E2=80=94=200?= =?UTF-8?q?.4608,=20below=20all-neutral=20baseline,=20do=20not=20adopt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the 2026-07-02 training run in the spec: fine-tuning beat the pretrained sentiment head by 9 points (real signal exists) but still lost to always-predict-neutral (0.516) by 5.5 points, with immediate overfitting past epoch 1. Verdict per the spec's own success criteria: do not adopt. Recommendations: 0.516 is the honest baseline, remaining levers are headline aggregation / non-text features, and the 0.60 target needs renegotiating with the team. --- .../2026-07-02-finbert-finetune-design.md | 55 ++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md index f5137d3..5238045 100644 --- a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md +++ b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md @@ -2,7 +2,7 @@ **Date:** 2026-07-02 **Owner:** Jack (proposal) — implementation lands in Nadi's classifier, so **requires Nadi's sign-off**; the split change touches Aurora's output, so **requires Aurora's sign-off** too. -**Status:** proposed — awaiting review +**Status:** implemented and run — **verdict: do not adopt** (results at bottom). Owner review of the Aurora/Nadi changes still pending. ## Problem @@ -150,3 +150,56 @@ handoff chain; only Nadi's generated script reads the model dir. - Headline aggregation per ticker-day (multiple headlines → one prediction). - Non-text features (momentum, volume) — would need a different model head. - Per-iteration retraining driven by the retune loop. + +--- + +## Results (2026-07-02 run) + +Trained as specced, option (c): dataset ended at 2019-12-31, time-based 80/20 cut at +**2019-08-15** → 7,991 train / 880 val (Jun–Aug 2019) / 2,196 test (Aug–Dec 2019, +pre-COVID). 3 epochs, batch 16, lr 2e-5, seed 42, ~12 min on M-series MPS. Full +record: `outputs/finetune_report.json`. + +| Model | Test accuracy | +|---|---| +| Pretrained FinBERT sentiment (prior runs) | 0.37 | +| **Fine-tuned FinBERT (this run)** | **0.4608** | +| All-neutral baseline (this test window) | **0.5159** | + +Per-class (fine-tuned): neutral **0.70**, down **0.34**, up **0.10**. + +| Epoch | Train loss | Val accuracy | +|---|---|---| +| 1 | 1.1158 | **0.4205** ← saved checkpoint | +| 2 | 1.0756 | 0.3693 | +| 3 | 1.0314 | 0.3648 | + +### Interpretation + +1. **Fine-tuning did extract real signal** — +9 points over the pretrained sentiment + head. The zero-signal finding was about *sentiment*, and training on the move + labels found some non-sentiment textual signal. +2. **But the model still loses to "always predict neutral"** by 5.5 points on the + held-out window. The `up` class is essentially never right (0.10). +3. **Overfitting was immediate**: validation peaked at epoch 1 and degraded while + train loss kept falling. More epochs / more tuning will not close a 5.5-point gap + to the majority baseline; the ceiling is the data, not the optimizer. + +### Decision (per §4 success criteria) + +Test accuracy **< all-neutral baseline → do not adopt.** The measured honest ceiling +of one-headline → next-day-direction on this dataset is ~0.46–0.52, and the 0.60 +target is not reachable with this task formulation. + +**Recommendations to the team:** + +- Treat **0.516 (all-neutral)** as the honest baseline any future model must beat. +- If the project continues on prediction quality: headline **aggregation per + ticker-day** and/or **non-text features** (momentum, volume) are the remaining + levers — both are new increments, not tweaks. +- Otherwise: **renegotiate the 0.60 target** — the pipeline, contracts, retune loop, + and evaluation machinery all work as designed; the target was set before anyone + measured whether the task supports it. +- `outputs/finbert_finetuned/` is deliberately left in place for inspection, but per + the don't-adopt verdict it should be moved aside (or deleted) before the next + pipeline run, because Nadi's generated classifier auto-prefers it when present. From 84c0a8bc3bd6587b24d4e34fbd4fbdbaee879b6f Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:19:11 +0200 Subject: [PATCH 4/8] docs: note rejected weights moved out of the auto-load path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit outputs/finbert_finetuned/ -> outputs/finbert_finetuned_rejected/ so Nadi's generated classifier no longer auto-prefers the rejected model; fallback to the pretrained sentiment head verified. (outputs/ is gitignored — only the spec note changes.) --- .../specs/2026-07-02-finbert-finetune-design.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md index 5238045..3c08579 100644 --- a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md +++ b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md @@ -200,6 +200,7 @@ target is not reachable with this task formulation. - Otherwise: **renegotiate the 0.60 target** — the pipeline, contracts, retune loop, and evaluation machinery all work as designed; the target was set before anyone measured whether the task supports it. -- `outputs/finbert_finetuned/` is deliberately left in place for inspection, but per - the don't-adopt verdict it should be moved aside (or deleted) before the next - pipeline run, because Nadi's generated classifier auto-prefers it when present. +- Per the don't-adopt verdict the weights were moved to + `outputs/finbert_finetuned_rejected/` (kept for inspection): Nadi's generated + classifier auto-prefers `outputs/finbert_finetuned/` when present, so with the dir + gone the next pipeline run falls back to the pretrained sentiment head (verified). From 5988542a8f31e7f07dfd041ef51a85beb44b7cb5 Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:22:29 +0200 Subject: [PATCH 5/8] =?UTF-8?q?docs:=20fine-tuned=20weights=20restored=20?= =?UTF-8?q?=E2=80=94=20pipeline=20uses=20them=20at=20Jack's=20direction?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Overrides the don't-adopt default for now: 0.46 beats the pretrained 0.37 even though both are below the all-neutral 0.516 baseline. The spec keeps the verdict and the fallback instructions. --- .../specs/2026-07-02-finbert-finetune-design.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md index 3c08579..9aec240 100644 --- a/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md +++ b/docs/superpowers/specs/2026-07-02-finbert-finetune-design.md @@ -200,7 +200,8 @@ target is not reachable with this task formulation. - Otherwise: **renegotiate the 0.60 target** — the pipeline, contracts, retune loop, and evaluation machinery all work as designed; the target was set before anyone measured whether the task supports it. -- Per the don't-adopt verdict the weights were moved to - `outputs/finbert_finetuned_rejected/` (kept for inspection): Nadi's generated - classifier auto-prefers `outputs/finbert_finetuned/` when present, so with the dir - gone the next pipeline run falls back to the pretrained sentiment head (verified). +- Weights were briefly moved aside per the verdict, then **restored to + `outputs/finbert_finetuned/` at Jack's direction** — pipeline runs use the + fine-tuned model for now (it beats pretrained 0.46 vs 0.37, even though both sit + below the all-neutral 0.516 baseline). Delete/move the dir to fall back to the + pretrained sentiment head; the generated classifier picks whichever is present. From 2be1db148c111e78b2a62a1f5e1b8ab6100c8ff7 Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:29:53 +0200 Subject: [PATCH 6/8] feat(pipeline): --dataset-end flag so runs can pin a pre-COVID window MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit main.py re-runs Aurora inside the pipeline, and without dataset_end it regenerated processed_data.csv with all rows — putting the 80/20 date cut's test window inside the Feb-Jun 2020 COVID crash, a regime the fine-tuned model never saw (run scored 0.37 with down-class accuracy 0.03 vs 0.46 on its pre-COVID window). Thread dataset_end from the CLI through pipeline_graph.run/build_pipeline to Aurora so `uv run main.py --dataset-end 2019-12-31` reproduces the training-time window. Also isolates the finetuned-model template test from the repo's real outputs/ dir (it failed once genuine weights existed there). --- agents/pipeline_graph.py | 16 +++++++++++----- main.py | 7 ++++++- tests/test_classifier.py | 4 ++++ 3 files changed, 21 insertions(+), 6 deletions(-) diff --git a/agents/pipeline_graph.py b/agents/pipeline_graph.py index 1b5496a..0cb7604 100644 --- a/agents/pipeline_graph.py +++ b/agents/pipeline_graph.py @@ -107,17 +107,22 @@ def build(cls, *, target_accuracy=0.60, max_iterations=5, patience=2, ) -def build_pipeline(agents: Agents, *, threshold=0.01, data_dir=None, checkpointer=None): +def build_pipeline(agents: Agents, *, threshold=0.01, data_dir=None, dataset_end=None, + checkpointer=None): """Compile the unified pipeline graph. `agents` supplies the five agents (real or fake); `threshold` is Aurora's labelling band; `data_dir` overrides where - Aurora reads fnspid_raw.csv (defaults to the repo `data/`). The node functions - close over these, so no non-serialisable objects live in the graph state. + Aurora reads fnspid_raw.csv (defaults to the repo `data/`); `dataset_end` + (YYYY-MM-DD) drops later rows before Aurora's train/test split. The node + functions close over these, so no non-serialisable objects live in the graph + state. """ from langgraph.graph import StateGraph, START, END def process(state: PipelineState) -> dict: """Aurora: build the labelled dataset. Runs once, before the loop.""" extra = {"data_dir": data_dir} if data_dir else {} + if dataset_end: + extra["dataset_end"] = dataset_end processed = agents.aurora.run(threshold=threshold, **extra)["processed_data_path"] return {"processed_data_path": processed} @@ -178,7 +183,8 @@ def route(state: PipelineState) -> str: def run(*, threshold=0.01, target_accuracy=0.60, max_iterations=5, patience=2, - min_delta=0.01, sample_size=300, use_ollama=True, data_dir=None) -> dict: + min_delta=0.01, sample_size=300, use_ollama=True, data_dir=None, + dataset_end=None) -> dict: """Build the pipeline with real agents and run it once end to end, returning the final graph state. This is the entry point `main.py` calls.""" from langgraph.checkpoint.memory import MemorySaver @@ -187,7 +193,7 @@ def run(*, threshold=0.01, target_accuracy=0.60, max_iterations=5, patience=2, patience=patience, min_delta=min_delta, sample_size=sample_size, use_ollama=use_ollama) graph = build_pipeline(agents, threshold=threshold, data_dir=data_dir, - checkpointer=MemorySaver()) + dataset_end=dataset_end, checkpointer=MemorySaver()) return graph.invoke( {"retune_request_path": None}, {"configurable": {"thread_id": "pipeline"}, "recursion_limit": RECURSION_LIMIT}, diff --git a/main.py b/main.py index 8f26605..a32d42e 100644 --- a/main.py +++ b/main.py @@ -40,13 +40,18 @@ def main(): p.add_argument("--sample-size", type=int, default=300, help="rows sampled for explanation") p.add_argument("--no-ollama", action="store_true", help="force Freddi's offline fallback") p.add_argument("--data-dir", default=None, help="override dir holding fnspid_raw.csv") + p.add_argument("--dataset-end", default=None, metavar="YYYY-MM-DD", + help="drop rows after this date before splitting (e.g. 2019-12-31 " + "keeps the COVID regime out of the test window, matching the " + "fine-tuned model's training data)") args = p.parse_args() load_dotenv() # make .env secrets (e.g. HF_TOKEN) visible to the agents final = run(threshold=args.threshold, target_accuracy=args.target_accuracy, max_iterations=args.max_iterations, patience=args.patience, min_delta=args.min_delta, sample_size=args.sample_size, - use_ollama=not args.no_ollama, data_dir=args.data_dir) + use_ollama=not args.no_ollama, data_dir=args.data_dir, + dataset_end=args.dataset_end) print(f"\n[main] done -- {final['final_action']} at iteration " f"{final['iteration']}. Outputs in {OUT}/") diff --git a/tests/test_classifier.py b/tests/test_classifier.py index d9f299c..2d78761 100644 --- a/tests/test_classifier.py +++ b/tests/test_classifier.py @@ -113,8 +113,12 @@ def test_generated_code_prefers_finetuned_model_when_present(outdir, monkeypatch """generate_code points the script at outputs/finbert_finetuned when the dir exists, and falls back to the pretrained hub model (MODEL_DIR = None) when it doesn't. Template-level check — no model download.""" + import agents.nadi_classifier as nc from agents.nadi_classifier import generate_code + # Isolate from the repo's real outputs/ (a genuine fine-tuned dir may exist there) + monkeypatch.setattr(nc, "OUTPUT_DIR", str(outdir)) + # No fine-tuned dir -> fallback code_path = outdir / "classifier.py" res = generate_code({"classifier_code_path": str(code_path)}) From f29a14a6e3dd61f20609919b09ca890cfccc6650 Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:37:17 +0200 Subject: [PATCH 7/8] feat(pipeline): finalize on best iteration, not last Accuracy can regress across retune passes (e.g. 0.39 at iter 3 but 0.37 at the forced proceed), yet each pass overwrote predictions/eval/classifier, so explain + finalize ran on the regressed artifacts. - evaluate node snapshots PREDS/EVAL/CODE to outputs/best/ on each new best - new select_best node on the proceed branch restores that snapshot and redraws sample_for_explanation.csv when the last iteration wasn't the best - extract Manager's sampling into shared write_sample() so the sample format keeps a single definition Gate semantics unchanged: the Manager still decides from the last iteration's report; restore happens after the verdict. --- agents/jack_manager.py | 21 ++++++--- agents/pipeline_graph.py | 57 ++++++++++++++++++++++--- tests/test_pipeline_graph.py | 82 ++++++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+), 13 deletions(-) diff --git a/agents/jack_manager.py b/agents/jack_manager.py index 50805a9..eb392f1 100644 --- a/agents/jack_manager.py +++ b/agents/jack_manager.py @@ -235,21 +235,30 @@ def write_retune(state: ManagerState) -> dict: return {"decision_log": [f"iteration {state['iteration']}: wrote retune_request.json"]} -def proceed(state: ManagerState) -> dict: - """sample_for_explanation.csv (Handoff 4) — drawn from predictions_test.csv. - Terminal: hands off to Freddi and waits for explanations.csv.""" +def write_sample(predictions_path: str, sample_size: int = 300) -> int: + """Write sample_for_explanation.csv (Handoff 4) from a predictions file. + Shared by the proceed node and the pipeline's best-iteration restore, so the + sample format has exactly one definition.""" import pandas as pd - preds = pd.read_csv(state.get("predictions_path", "mock_data/predictions_test.csv")) + preds = pd.read_csv(predictions_path) sample = (preds[["article_id", "article_title", "predicted_label", "label", "confidence", "prob_up", "prob_down", "prob_neutral"]] .rename(columns={"label": "actual_label"})) - n = min(len(sample), state.get("sample_size", 300)) + n = min(len(sample), sample_size) if n < len(sample): # only subsample when needed sample = sample.sample(n=n, random_state=42) # representative + reproducible - _write_decision(state) os.makedirs(OUTPUT_DIR, exist_ok=True) sample.to_csv(os.path.join(OUTPUT_DIR, "sample_for_explanation.csv"), index=False) + return n + + +def proceed(state: ManagerState) -> dict: + """sample_for_explanation.csv (Handoff 4) — drawn from predictions_test.csv. + Terminal: hands off to Freddi and waits for explanations.csv.""" + _write_decision(state) + n = write_sample(state.get("predictions_path", "mock_data/predictions_test.csv"), + state.get("sample_size", 300)) return {"decision_log": [f"iteration {state['iteration']}: wrote sample_for_explanation.csv ({n} rows)"]} diff --git a/agents/pipeline_graph.py b/agents/pipeline_graph.py index 0cb7604..bbe8101 100644 --- a/agents/pipeline_graph.py +++ b/agents/pipeline_graph.py @@ -20,7 +20,12 @@ ## Graph shape START → process → classify → evaluate → gate ─(retune)→ classify [CYCLE] - └(proceed)→ explain → finalize → END + └(proceed)→ select_best → explain → finalize → END + +`evaluate` snapshots each new-best iteration's artifacts into `outputs/best/`; +`select_best` restores that snapshot (and redraws the explanation sample) when the +loop's LAST iteration wasn't its best — accuracy can regress across retunes, and +without this the pipeline would finalize on the regressed predictions. `gate` is the Manager: calling it writes either `retune_request.json` (retune) or `sample_for_explanation.csv` (proceed). Its own checkpointer carries the iteration @@ -36,7 +41,9 @@ from __future__ import annotations +import json import os +import shutil from dataclasses import dataclass from typing_extensions import TypedDict @@ -46,13 +53,13 @@ from agents.nadi_classifier import ClassifierAgent from agents.sabina_evaluator import EvaluatorAgent from agents.freddi_explanation import ExplanationAgent - from agents.jack_manager import ManagerAgent, OUTPUT_DIR as OUT + from agents.jack_manager import ManagerAgent, write_sample, OUTPUT_DIR as OUT except ModuleNotFoundError: # running as a bare script with agents/ on sys.path from aurora_processing import ProcessingAgent from nadi_classifier import ClassifierAgent from sabina_evaluator import EvaluatorAgent from freddi_explanation import ExplanationAgent - from jack_manager import ManagerAgent, OUTPUT_DIR as OUT + from jack_manager import ManagerAgent, write_sample, OUTPUT_DIR as OUT # Contract files exchanged between the nodes. All live under the Manager's # OUTPUT_DIR except processed_data.csv, whose path Aurora returns at runtime. @@ -63,6 +70,10 @@ RETUNE = os.path.join(OUT, "retune_request.json") EXPL = os.path.join(OUT, "explanations.csv") +# Snapshot dir for the best iteration's artifacts (internal to the loop, not a +# contract handoff). Holds copies of CODE/PREDS/EVAL from the highest-accuracy pass. +BEST_DIR = os.path.join(OUT, "best") + # A retune cycle is 3 nodes (classify → evaluate → gate); with the process/explain/ # finalize tail, ~5 iterations stays well under this. LangGraph aborts a runaway # cycle at recursion_limit, so we set headroom rather than rely on the default 25. @@ -78,6 +89,8 @@ class PipelineState(TypedDict, total=False): retune_request_path: str | None # None on the first pass; RETUNE on cycle passes final_action: str # the gate's verdict: "retune" | "proceed" iteration: int # Manager's loop counter (for reporting) + last_accuracy: float # accuracy of the most recent evaluate pass + best_accuracy: float # best accuracy seen across cycle passes @dataclass @@ -136,9 +149,19 @@ def classify(state: PipelineState) -> dict: return {} def evaluate(state: PipelineState) -> dict: - """Sabina: score the predictions and write evaluation_report.json.""" + """Sabina: score the predictions and write evaluation_report.json. Also + snapshots this pass's artifacts into BEST_DIR whenever accuracy sets a new + best, so `select_best` can restore them if later retunes regress.""" agents.sabina.run(predictions=PREDS, classifier_code=CODE) - return {} + with open(EVAL, encoding="utf-8") as f: + acc = json.load(f)["accuracy"] + out = {"last_accuracy": acc} + if acc > state.get("best_accuracy", -1.0): + os.makedirs(BEST_DIR, exist_ok=True) + for path in (PREDS, EVAL, CODE): + shutil.copy2(path, os.path.join(BEST_DIR, os.path.basename(path))) + out["best_accuracy"] = acc + return out def gate(state: PipelineState) -> dict: """Manager gate. Invoking it applies the accuracy gate (with convergence + @@ -152,6 +175,24 @@ def gate(state: PipelineState) -> dict: out["retune_request_path"] = RETUNE # fed back into `classify` on the cycle return out + def select_best(state: PipelineState) -> dict: + """The gate proceeded with the LAST iteration's artifacts, which are not + necessarily the best ones (accuracy can regress across retunes). If an + earlier pass scored higher, restore its snapshot over the canonical paths + and redraw the explanation sample from the restored predictions, so + explain/finalize run on the best iteration. Gate semantics are untouched: + the Manager already made its verdict from the last iteration's report.""" + best, last = state.get("best_accuracy", -1.0), state.get("last_accuracy", -1.0) + if best <= last: + return {} + for path in (PREDS, EVAL, CODE): + shutil.copy2(os.path.join(BEST_DIR, os.path.basename(path)), path) + sample_size = getattr(agents.manager, "_defaults", {}).get("sample_size", 300) + write_sample(PREDS, sample_size) + print(f"[pipeline] last iteration regressed ({last:.2f}) — restored best " + f"iteration's artifacts ({best:.2f}) for explanation + finals") + return {} + def explain(state: PipelineState) -> dict: """Freddi: justify each sampled prediction into explanations.csv.""" agents.freddi.run(sample_for_explanation=SAMPLE, output=EXPL) @@ -170,13 +211,15 @@ def route(state: PipelineState) -> str: b = StateGraph(PipelineState) for name, fn in [("process", process), ("classify", classify), ("evaluate", evaluate), - ("gate", gate), ("explain", explain), ("finalize", finalize)]: + ("gate", gate), ("select_best", select_best), ("explain", explain), + ("finalize", finalize)]: b.add_node(name, fn) b.add_edge(START, "process") b.add_edge("process", "classify") b.add_edge("classify", "evaluate") b.add_edge("evaluate", "gate") - b.add_conditional_edges("gate", route, {"retune": "classify", "proceed": "explain"}) + b.add_conditional_edges("gate", route, {"retune": "classify", "proceed": "select_best"}) + b.add_edge("select_best", "explain") b.add_edge("explain", "finalize") b.add_edge("finalize", END) return b.compile(checkpointer=checkpointer) diff --git a/tests/test_pipeline_graph.py b/tests/test_pipeline_graph.py index 65ee882..fa3a368 100644 --- a/tests/test_pipeline_graph.py +++ b/tests/test_pipeline_graph.py @@ -66,6 +66,88 @@ def run(self, *, predictions, classifier_code): return {"output_path": "outputs/evaluation_report.json"} +class MarkedNadi: + """FakeNadi variant that stamps every predictions file with the pass number, + so the test can tell WHICH iteration's artifacts survived to the end.""" + def __init__(self): + self.calls = 0 + + def run(self, *, processed_data, classifier_code, predictions, retune_request=None): + self.calls += 1 + Path(predictions).parent.mkdir(parents=True, exist_ok=True) + shutil.copy(MOCK_PRED, predictions) + import pandas as pd + df = pd.read_csv(predictions) + df["fake_pass"] = self.calls + df.to_csv(predictions, index=False) + Path(classifier_code).write_text(f"THRESHOLD = 0.5 # pass {self.calls}\n") + return {} + + +class PeakSabina: + """Accuracy peaks on pass 2 then regresses, so the best iteration is NOT the + last one the gate proceeds with — exactly the case select_best must fix.""" + ACCS = [0.20, 0.39, 0.30, 0.30, 0.30, 0.30, 0.30, 0.30] + + def __init__(self): + self.calls = 0 + + def run(self, *, predictions, classifier_code): + acc = self.ACCS[self.calls] + self.calls += 1 + report = { + "accuracy": acc, + "below_threshold": True, + "class_accuracy": {"up": 0.3, "down": 0.2, "neutral": 0.5}, + "misclassified_ids": [], + "proposal": {"recommended_action": "retune", "reason": "low", + "focus_labels": ["down"], + "suggested_params": {"threshold": 0.5, "max_length": 128}, + "code_notes": ""}, + } + os.makedirs("outputs", exist_ok=True) + Path("outputs/evaluation_report.json").write_text(json.dumps(report)) + return {"output_path": "outputs/evaluation_report.json"} + + +@needs_langgraph +def test_best_iteration_restored_when_accuracy_regresses(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "outputs").mkdir() + + import pandas as pd + import agents.pipeline_graph as pg + from agents.jack_manager import ManagerAgent + from agents.freddi_explanation import ExplanationAgent + from langgraph.checkpoint.memory import MemorySaver + + nadi, sabina = MarkedNadi(), PeakSabina() + agents = pg.Agents( + aurora=FakeAurora(), nadi=nadi, sabina=sabina, + manager=ManagerAgent(predictions_path=pg.PREDS, target_accuracy=0.60, + max_iterations=6, patience=2, min_delta=0.01), + freddi=ExplanationAgent(use_ollama=False, output_path=pg.EXPL), + ) + graph = pg.build_pipeline(agents, checkpointer=MemorySaver()) + final = graph.invoke({"retune_request_path": None}, + {"configurable": {"thread_id": "t"}, "recursion_limit": pg.RECURSION_LIMIT}) + + assert final["final_action"] == "proceed" + assert nadi.calls >= 3, "loop must run past the accuracy peak for this test to bite" + + # The artifacts that survive are the BEST pass's (accuracy 0.39 = pass 2), + # not the last pass's — the whole point of select_best. + preds = pd.read_csv(tmp_path / "outputs" / "predictions_test.csv") + assert preds["fake_pass"].iloc[0] == 2, "final predictions should come from the best pass" + report = json.loads((tmp_path / "outputs" / "evaluation_report.json").read_text()) + assert report["accuracy"] == 0.39 + final_report = json.loads((tmp_path / "outputs" / "final_report.json").read_text()) + assert final_report["final_accuracy"] == 0.39 + + # The explanation sample was redrawn from the restored predictions. + assert (tmp_path / "outputs" / "sample_for_explanation.csv").exists() + + @needs_langgraph def test_graph_cycles_then_finalizes(tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) From e88f7e349c050460c68f203f2893d82948dab54f Mon Sep 17 00:00:00 2001 From: Nikolas Jack Altran Date: Thu, 2 Jul 2026 15:50:21 +0200 Subject: [PATCH 8/8] Revert "feat(pipeline): --dataset-end flag so runs can pin a pre-COVID window" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverts the --dataset-end CLI plumbing from 2be1db1 (main.py + pipeline_graph.run/build_pipeline). Keeps that commit's test isolation fix in tests/test_classifier.py. Aurora's dataset_end param itself stays — still reachable via ProcessingAgent.run(dataset_end=...). --- agents/pipeline_graph.py | 16 +++++----------- main.py | 7 +------ 2 files changed, 6 insertions(+), 17 deletions(-) diff --git a/agents/pipeline_graph.py b/agents/pipeline_graph.py index 0cb7604..1b5496a 100644 --- a/agents/pipeline_graph.py +++ b/agents/pipeline_graph.py @@ -107,22 +107,17 @@ def build(cls, *, target_accuracy=0.60, max_iterations=5, patience=2, ) -def build_pipeline(agents: Agents, *, threshold=0.01, data_dir=None, dataset_end=None, - checkpointer=None): +def build_pipeline(agents: Agents, *, threshold=0.01, data_dir=None, checkpointer=None): """Compile the unified pipeline graph. `agents` supplies the five agents (real or fake); `threshold` is Aurora's labelling band; `data_dir` overrides where - Aurora reads fnspid_raw.csv (defaults to the repo `data/`); `dataset_end` - (YYYY-MM-DD) drops later rows before Aurora's train/test split. The node - functions close over these, so no non-serialisable objects live in the graph - state. + Aurora reads fnspid_raw.csv (defaults to the repo `data/`). The node functions + close over these, so no non-serialisable objects live in the graph state. """ from langgraph.graph import StateGraph, START, END def process(state: PipelineState) -> dict: """Aurora: build the labelled dataset. Runs once, before the loop.""" extra = {"data_dir": data_dir} if data_dir else {} - if dataset_end: - extra["dataset_end"] = dataset_end processed = agents.aurora.run(threshold=threshold, **extra)["processed_data_path"] return {"processed_data_path": processed} @@ -183,8 +178,7 @@ def route(state: PipelineState) -> str: def run(*, threshold=0.01, target_accuracy=0.60, max_iterations=5, patience=2, - min_delta=0.01, sample_size=300, use_ollama=True, data_dir=None, - dataset_end=None) -> dict: + min_delta=0.01, sample_size=300, use_ollama=True, data_dir=None) -> dict: """Build the pipeline with real agents and run it once end to end, returning the final graph state. This is the entry point `main.py` calls.""" from langgraph.checkpoint.memory import MemorySaver @@ -193,7 +187,7 @@ def run(*, threshold=0.01, target_accuracy=0.60, max_iterations=5, patience=2, patience=patience, min_delta=min_delta, sample_size=sample_size, use_ollama=use_ollama) graph = build_pipeline(agents, threshold=threshold, data_dir=data_dir, - dataset_end=dataset_end, checkpointer=MemorySaver()) + checkpointer=MemorySaver()) return graph.invoke( {"retune_request_path": None}, {"configurable": {"thread_id": "pipeline"}, "recursion_limit": RECURSION_LIMIT}, diff --git a/main.py b/main.py index a32d42e..8f26605 100644 --- a/main.py +++ b/main.py @@ -40,18 +40,13 @@ def main(): p.add_argument("--sample-size", type=int, default=300, help="rows sampled for explanation") p.add_argument("--no-ollama", action="store_true", help="force Freddi's offline fallback") p.add_argument("--data-dir", default=None, help="override dir holding fnspid_raw.csv") - p.add_argument("--dataset-end", default=None, metavar="YYYY-MM-DD", - help="drop rows after this date before splitting (e.g. 2019-12-31 " - "keeps the COVID regime out of the test window, matching the " - "fine-tuned model's training data)") args = p.parse_args() load_dotenv() # make .env secrets (e.g. HF_TOKEN) visible to the agents final = run(threshold=args.threshold, target_accuracy=args.target_accuracy, max_iterations=args.max_iterations, patience=args.patience, min_delta=args.min_delta, sample_size=args.sample_size, - use_ollama=not args.no_ollama, data_dir=args.data_dir, - dataset_end=args.dataset_end) + use_ollama=not args.no_ollama, data_dir=args.data_dir) print(f"\n[main] done -- {final['final_action']} at iteration " f"{final['iteration']}. Outputs in {OUT}/")