FinBERT fine-tuning on move labels: spec, implementation, results#27
Closed
njaltran wants to merge 9 commits into
Closed
FinBERT fine-tuning on move labels: spec, implementation, results#27njaltran wants to merge 9 commits into
njaltran wants to merge 9 commits into
Conversation
Grounds the proposal in the 2026-07-02 zero-signal analysis (sentiment uncorrelated with next-day move, model below the all-neutral baseline) and specs the only ceiling-changing option: fine-tune FinBERT on the move labels with a leak-free time-based split. Covers the COVID regime-shift problem in the natural 80/20 date cut, contract impact (split column in processed_data.csv), fallback behavior, and the sign-offs needed from Aurora and Nadi per AGENTS.md.
Implements docs/superpowers/specs/2026-07-02-finbert-finetune-design.md — the only ceiling-changing option after the zero-signal finding (pretrained sentiment is uncorrelated with next-day move). - Aurora: `split` column in processed_data.csv — time-based 80/20 date cut (train = earliest dates) so no future info leaks into training; optional `dataset_end` state param (e.g. "2019-12-31") keeps the COVID regime out of the test window. Extracted as _assign_split(). - New agents/finetune_finbert.py: standalone script, plain torch loop (no new deps), weighted CE for the neutral-heavy distribution, time-ordered validation slice, keeps best-val weights, writes outputs/finbert_finetuned/ + finetune_report.json. - Nadi template: MODEL_DIR support — loads fine-tuned weights when outputs/finbert_finetuned/ exists (id2label already up/down/neutral), falls back to the pretrained sentiment head otherwise; predicts only split=test rows when the input has a split column. - Tests: split monotonicity, dataset_end, val ordering, template fallback/preference, test-rows-only predictions, opt-in `slow` smoke train (pytest -m slow). Contract doc + mock_data updated. Touches Aurora's and Nadi's modules per the spec's sign-off table — implemented ahead of sign-off at Jack's direction; owners should review before this merges.
…adopt Records the 2026-07-02 training run in the spec: fine-tuning beat the pretrained sentiment head by 9 points (real signal exists) but still lost to always-predict-neutral (0.516) by 5.5 points, with immediate overfitting past epoch 1. Verdict per the spec's own success criteria: do not adopt. Recommendations: 0.516 is the honest baseline, remaining levers are headline aggregation / non-text features, and the 0.60 target needs renegotiating with the team.
outputs/finbert_finetuned/ -> outputs/finbert_finetuned_rejected/ so Nadi's generated classifier no longer auto-prefers the rejected model; fallback to the pretrained sentiment head verified. (outputs/ is gitignored — only the spec note changes.)
…ction Overrides the don't-adopt default for now: 0.46 beats the pretrained 0.37 even though both are below the all-neutral 0.516 baseline. The spec keeps the verdict and the fallback instructions.
There was a problem hiding this comment.
Pull request overview
This PR adds a leak-resistant, time-based train/test split to processed_data.csv, introduces a standalone FinBERT fine-tuning script for up/down/neutral labels, and updates the classifier generation to prefer fine-tuned weights (with fallback), along with updated contracts, mock data, and tests documenting the measured accuracy ceiling.
Changes:
- Add Aurora-side
splitgeneration (and optionaldataset_end) to prevent future leakage in fine-tuning/evaluation. - Add
agents/finetune_finbert.pyto fine-tune FinBERT on move labels and emit a reproducibility report. - Update Nadi’s classifier template/codegen to load fine-tuned weights when present and predict only
split=testrows; expand tests and pytest config for opt-in slow training.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
agents/aurora_processing.py |
Adds time-based split column generation and optional dataset_end filtering before export. |
agents/finetune_finbert.py |
New standalone fine-tuning script with train/val/test splitting, training loop, and report output. |
agents/nadi_classifier.py |
Updates generated classifier to prefer fine-tuned weights and restrict predictions to split=test rows when available. |
agents/state.py |
Extends pipeline state with optional dataset_end parameter. |
docs/data_contracts.md |
Updates Handoff 1 and Handoff 2 contracts to include/describe split behavior. |
docs/superpowers/specs/2026-07-02-finbert-finetune-design.md |
Adds full design spec + run results and decision guidance. |
mock_data/processed_data.csv |
Updates mock processed data to include the new split column. |
pyproject.toml |
Adds slow marker and defaults pytest to exclude slow tests. |
tests/test_classifier.py |
Adds tests for fine-tuned model preference and enforcing predictions over test split only. |
tests/test_finetune_split.py |
Adds tests for split monotonicity, dataset end filtering, time-ordered validation, and an opt-in smoke fine-tune. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+149
to
+154
| # Prefer fine-tuned weights when a training run has produced them (see | ||
| # agents/finetune_finbert.py); otherwise the generated script falls back to | ||
| # the pretrained sentiment head. | ||
| model_dir = state.get("model_dir") or os.path.join(OUTPUT_DIR, "finbert_finetuned") | ||
| if not os.path.isdir(model_dir): | ||
| model_dir = None |
Comment on lines
+88
to
+90
| split_cut = pd.to_datetime(df["date"]).quantile(0.8) | ||
| df["split"] = ["train" if d <= split_cut else "test" | ||
| for d in pd.to_datetime(df["date"])] |
Comment on lines
+61
to
+65
| train_all = df[df["split"] == "train"] | ||
| test = df[df["split"] == "test"] | ||
| val_cut = pd.to_datetime(train_all["date"]).quantile(0.9) | ||
| is_val = pd.to_datetime(train_all["date"]) > val_cut | ||
| return train_all[~is_val], train_all[is_val], test |
Comment on lines
+75
to
+79
| for pred, true in zip(preds, batch["labels"]): | ||
| per_class[int(true)][1] += 1 | ||
| if pred == true: | ||
| per_class[int(true)][0] += 1 | ||
| correct_total += 1 |
Comment on lines
+75
to
+76
| - HF `Trainer` on `ProsusAI/finbert` with a fresh 3-label head | ||
| (`num_labels=3`, `ignore_mismatched_sizes=True`). |
main.py re-runs Aurora inside the pipeline, and without dataset_end it regenerated processed_data.csv with all rows — putting the 80/20 date cut's test window inside the Feb-Jun 2020 COVID crash, a regime the fine-tuned model never saw (run scored 0.37 with down-class accuracy 0.03 vs 0.46 on its pre-COVID window). Thread dataset_end from the CLI through pipeline_graph.run/build_pipeline to Aurora so `uv run main.py --dataset-end 2019-12-31` reproduces the training-time window. Also isolates the finetuned-model template test from the repo's real outputs/ dir (it failed once genuine weights existed there).
Accuracy can regress across retune passes (e.g. 0.39 at iter 3 but 0.37 at the forced proceed), yet each pass overwrote predictions/eval/classifier, so explain + finalize ran on the regressed artifacts. - evaluate node snapshots PREDS/EVAL/CODE to outputs/best/ on each new best - new select_best node on the proceed branch restores that snapshot and redraws sample_for_explanation.csv when the last iteration wasn't the best - extract Manager's sampling into shared write_sample() so the sample format keeps a single definition Gate semantics unchanged: the Manager still decides from the last iteration's report; restore happens after the verdict.
…D window" Reverts the --dataset-end CLI plumbing from 2be1db1 (main.py + pipeline_graph.run/build_pipeline). Keeps that commit's test isolation fix in tests/test_classifier.py. Aurora's dataset_end param itself stays — still reachable via ProcessingAgent.run(dataset_end=...).
feat(pipeline): finalize on best iteration, not last
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After PR #26's analysis showed pretrained FinBERT sentiment is statistically independent of next-day price move (corr −0.008; accuracy 0.37 ≈ chance; all-neutral baseline 0.467 beats it), fine-tuning on our actual
up/down/neutrallabels was the only option left that could change the accuracy ceiling. This PR specs it, implements it, runs it, and reports the measured ceiling.What
docs/superpowers/specs/2026-07-02-finbert-finetune-design.mdagents/aurora_processing.py):processed_data.csvgains a time-basedsplitcolumn (earliest 80% of dates = train) so no future information leaks into training; optionaldataset_endparam (we used2019-12-31to keep the COVID regime out of the test window)agents/finetune_finbert.py: standalone training script — plain torch loop (no new deps), fresh 3-label head, weighted cross-entropy, time-ordered validation slice, keeps best-validation weights, writesoutputs/finbert_finetuned/+finetune_report.jsonagents/nadi_classifier.py): generated classifier auto-loads fine-tuned weights whenoutputs/finbert_finetuned/exists, falls back to the pretrained sentiment head otherwise; predicts onlysplit=testrows when the input carries the split columndocs/data_contracts.mdHandoff 1 gainssplit;mock_data/processed_data.csvupdated to matchdataset_end, validation ordering, template fallback/preference, test-rows-only predictions, opt-inslowsmoke train). 33/33 default suite green.Results (full table in the spec)
Fine-tuning found real signal (+9 pts over sentiment) but still loses to always-predict-neutral by 5.5 pts;
upclass accuracy 0.10; overfits past epoch 1. Per the spec's own success criteria the verdict is do not adopt — the honest ceiling of one-headline → next-day-direction is ~0.46–0.52 and the 0.60 target is unreachable with this task formulation. Jack has opted to run the pipeline on the fine-tuned model for now anyway (it beats pretrained); deleteoutputs/finbert_finetuned/to fall back.For the team