Dashboard + adaptive retune loop fixes (Increments 1-2 follow-ups)#26
Merged
Conversation
- main.py drives all five agents: Aurora -> (Nadi -> Sabina -> Manager retune loop) -> Freddi -> Manager finalize. Passes the real predictions path so the Manager never falls back to mock data. - Nadi's generated classifier authenticates to the HF Hub via HF_TOKEN / HUGGING_FACE_HUB_TOKEN when present, unauthenticated otherwise. - Regenerate uv.lock (was unparseable) so uv sync/run work again.
Reactive dashboard reading the agents' contract files (outputs/, falling back to mock_data/): KPIs, confusion matrix, confidence histogram, per-ticker accuracy, filterable predictions/explanations tables, and a live node-by-node trace of the Manager's LangGraph state. Widgets: confidence-threshold slider, ticker multiselect, label dropdown, misclassified-only switch. Adds pyarrow for arrow-backed tables.
Header accuracy now computes straight from predictions_test.csv (so it can't drift from the slider/table), loop iterations come from decision.json, and the threshold gate from evaluation_report. Stops reading final_report.json, whose convergence numbers can lag the latest run.
Compute the correct/incorrect flag once on load and reuse it across KPI, charts, slider and table; hoist the label list into a LABELS constant shared by the confusion matrix and dropdown; simplify the widget and graph-stream cells. No behavior change.
- ManagerState gains accuracy_history + tried_params reducer fields. - decide() proceeds early when accuracy plateaus (patience/min_delta), not just at the iteration cap. - retune escalates hyperparameters from history instead of repeating a constant proposal; first retune accepts Sabina's proposal, later ones override with fresh params from a schedule. - ManagerAgent gains patience/min_delta config. New tests cover convergence, non-repeating params, and history accumulation.
- agents/pipeline_graph.py composes all five agents into one LangGraph whose retune loop is a real cycle (gate -> classify -> evaluate -> gate), replacing main.py's Python for-loop. Agents are invoked as nodes (not flattened) to keep per-agent isolation; injectable via an Agents bundle. - main.py becomes a thin CLI over the graph; adds --patience/--min-delta. - Offline behavioural test drives the cycle with fakes for the heavy agents. - README documents the cycle + adaptive/convergence loop.
_next_threshold() reads the classifier's current THRESHOLD and lowers it by 0.05 per retune (floored at 0.35), so each retune proposal explores a gate Nadi hasn't already run instead of repeating the same 0.5 value.
decision.json is overwritten every iteration, so the per-iteration accuracy trend was lost once the loop moved past iteration 1 — only the final iteration's accuracy survived on disk. Write the cumulative accuracy_history reducer field into decision.json each time so the full trend is recoverable after a run finishes. Updates the contract doc and mock_data to match.
focus_labels only flagged the single class tied for the exact min score, so a near-tied second-worst class (e.g. up at 0.30 next to down at 0.28) got no boost from Nadi's retune. Flag anything within FOCUS_MARGIN (0.05) of the weakest instead. Also widen _RETUNE_SCHEDULE from 4 to 6 steps with finer threshold/ max_length increments, so each retune explores a smaller, less disruptive change instead of jumping in coarse 0.05/64 steps.
Sabina's first-retune floor was 0.35 while jack_manager.py's _RETUNE_SCHEDULE (used on every retune after the first) goes down to 0.20 — so the first retune stopped exploring earlier than later ones would. Match the floor to 0.20.
The 1.25x boost Nadi applies to focus_labels' softmax probability was hardcoded, so it never escalated across retunes the way threshold and max_length already did. Thread boost_factor through suggested_params -> generate_code -> BOOST_FACTOR, defaulting to 1.25 when unset, and escalate it in jack_manager.py's _RETUNE_SCHEDULE (1.25 -> 1.75) alongside the existing threshold/max_length steps. Crosses into Nadi's agents/nadi_classifier.py — flagging per AGENTS.md ownership; happy to hand off if Nadi wants to take it from here. Updates docs/data_contracts.md and mock_data/retune_request.json to document the new suggested_params key.
Adds a follow-up section to the adaptive retune loop design doc covering the five fixes made after Increment 1 shipped but the run still plateaued at 0.37: threshold step-down, accuracy_history on disk, focus_labels margin widening, threshold floor alignment, and tunable boost_factor. Notes the boost_factor change crosses into Nadi's file and isn't yet signed off, and restates the ceiling caveat (none of this retrains FinBERT).
classifier.py is a single contract file that gets overwritten every
retune, so past iterations' generated code was unrecoverable once the
loop moved on — same problem accuracy_history/decision.json had.
generate_code now also writes a copy to
classifier_history/classifier_iter{N}.py, numbered by the retune
request's iteration (0 for the pre-retune first pass), and returns
classifier_history_path in state.
Crosses into Nadi's agents/nadi_classifier.py per AGENTS.md ownership
— flagging, not yet signed off.
The 2026-07-02 run wasted its second retune re-running an identical
classifier: Sabina's accepted first proposal {threshold: 0.45,
max_length: 128} has no boost_factor key, so plain `params not in
tried` saw schedule entry 0 ({threshold: 0.45, max_length: 128,
boost_factor: 1.25}) as untried — and 1.25 is Nadi's default, so the
generated code was byte-identical (caught by the classifier_history
archive). Compare schedule entries against tried sets on their shared
keys instead.
There was a problem hiding this comment.
Pull request overview
This PR extends the pipeline with a unified LangGraph-based orchestration graph and a Marimo dashboard, while tightening the adaptive retune loop behavior (threshold step-down, focus-label widening, new boost_factor, classifier code archiving, and decision/accuracy-history persistence) and updating related contracts and tests.
Changes:
- Implement unified cyclic pipeline graph (
gate → classify → evaluate → gate) and amain.pyCLI entrypoint to run it. - Improve adaptive retune loop: stepped threshold proposals, broadened focus-label selection, retune param dedupe via shared-key comparison, persisted
accuracy_history, and per-iteration classifier code archiving. - Add a Marimo dashboard + layouts and update data contracts/mock data + tests to cover the new behavior.
Reviewed changes
Copilot reviewed 20 out of 22 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_sabina_evaluator.py | Adds regression tests for stepped-down threshold proposals and floor behavior. |
| tests/test_pipeline_graph.py | Adds an integration test proving the unified graph cycles and then finalizes. |
| tests/test_manager.py | Adds convergence/retune-adaptation/history persistence tests and a shared-keys dedupe regression. |
| tests/test_classifier.py | Extends retune tests for boost_factor and adds per-iteration classifier archive assertions. |
| requirements.txt | Adds dashboard dependencies (marimo/altair/pyarrow). |
| README.md | Expands project overview and documents pipeline flow with a Mermaid diagram + run instructions. |
| pyproject.toml | Adds dashboard dependencies and pins minimum versions for uv-managed installs. |
| mock_data/retune_request.json | Updates example retune request with optional boost_factor. |
| mock_data/decision.json | Updates example decision record with cumulative accuracy_history. |
| main.py | Adds CLI entrypoint to run the unified pipeline graph and load a local .env. |
| layouts/dashboard.slides.json | Adds Marimo dashboard layout (slides). |
| layouts/dashboard.grid.json | Adds Marimo dashboard layout (grid). |
| docs/superpowers/specs/2026-07-01-adaptive-retune-loop-design.md | Adds detailed design/spec for convergence + adaptive retune (and follow-up fixes). |
| docs/data_contracts.md | Updates contracts to document accuracy_history, boost_factor, and classifier-history archiving. |
| dashboard.py | Adds Marimo dashboard for outputs + Manager trace visualization. |
| agents/state.py | Extends pipeline state with classifier_history_path. |
| agents/sabina_evaluator.py | Implements stepped-down threshold proposals, threshold floor alignment, and widened focus-label selection. |
| agents/pipeline_graph.py | Introduces the unified LangGraph pipeline with a true retune cycle. |
| agents/nadi_classifier.py | Adds boost_factor, HF token passthrough, and per-iteration classifier code archiving. |
| agents/jack_manager.py | Adds convergence early-stop, adaptive retune schedule + shared-key dedupe, and persists accuracy_history to decision.json. |
| .python-version | Pins runtime to Python 3.13. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| "decision_log": [note], # reducer field → appended | ||
| "notes": note, # plain field → overwrites | ||
| "decision_log": [note], # reducer field → appended | ||
| "accuracy_history": [accuracy], # reducer field → appended |
Comment on lines
+104
to
+128
| @app.cell | ||
| def _(LABELS, alt, mo, preds): | ||
| # --- Confusion matrix: actual (label) vs predicted_label --- | ||
| cm = preds.groupby(["label", "predicted_label"]).size().reset_index(name="count") | ||
| confusion = ( | ||
| alt.Chart(cm) | ||
| .mark_rect() | ||
| .encode( | ||
| x=alt.X("predicted_label:N", sort=LABELS, title="Predicted"), | ||
| y=alt.Y("label:N", sort=LABELS, title="Actual"), | ||
| color=alt.Color("count:Q", scale=alt.Scale(scheme="blues")), | ||
| tooltip=["label", "predicted_label", "count"], | ||
| ) | ||
| .properties(width=260, height=260, title="Confusion matrix") | ||
| ) | ||
| text = confusion.mark_text(baseline="middle").encode( | ||
| text="count:Q", | ||
| color=alt.condition( | ||
| alt.datum.count > cm["count"].max() / 2, | ||
| alt.value("white"), | ||
| alt.value("black"), | ||
| ), | ||
| ) | ||
| mo.ui.altair_chart(confusion + text) if not preds.empty else mo.md("_no predictions_") | ||
| return |
Comment on lines
+295
to
+317
| from agents.jack_manager import ManagerAgent | ||
|
|
||
| with open(report_path, encoding="utf-8") as f: | ||
| report = json.load(f) | ||
|
|
||
| mgr = ManagerAgent(thread_id="dashboard") | ||
| init_state = {**mgr._defaults, "evaluation_report": report} | ||
|
|
||
| # One row per node firing: which keys it set and the headline values. | ||
| trace_rows = [ | ||
| { | ||
| "node": node, | ||
| "keys_updated": ", ".join(update.keys()), | ||
| "final_action": update.get("final_action", ""), | ||
| "decision": update.get("decision", ""), | ||
| "iteration": update.get("iteration", ""), | ||
| "notes": update.get("notes", ""), | ||
| } | ||
| for event in mgr._graph.stream(init_state, mgr._config) | ||
| for node, update in event.items() | ||
| ] | ||
| graph_trace = pd.DataFrame(trace_rows) | ||
| final_state = mgr._graph.get_state(mgr._config).values |
Comment on lines
+286
to
+288
| # Stream the Manager graph and capture each node's state update. Read-only | ||
| # re-run against the current report — writes go to outputs/ exactly as a | ||
| # normal run would, so guard on the report existing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
20 commits: dashboard work, the unified cyclic pipeline graph (Increment 2), convergence early-stop + adaptive retune (Increment 1), and this week's retune-loop follow-up fixes:
_next_threshold(): Sabina's retune proposal steps the classifier's current threshold down instead of always proposing 0.5decision.jsonnow persists cumulativeaccuracy_history(was lost to per-iteration overwrite)focus_labelswidened to classes within 0.05 of the weakest (previously only the exact-min class got boosted)THRESHOLD_FLOORaligned with the Manager schedule floor (0.20)boost_factor, escalated 1.25→1.75 in the schedule)classifier_history/classifier_iterN.pyHeads-up for owners
Per AGENTS.md ownership: this touches Sabina's
agents/sabina_evaluator.pyand Nadi'sagents/nadi_classifier.py(flagged in the individual commits).docs/data_contracts.md+mock_data/updated in the same changes:suggested_paramsgains optionalboost_factor,decision.jsongainsaccuracy_history.Analysis note
Deep-dive on the flat 0.37 accuracy: FinBERT sentiment is statistically independent of next-day move on this dataset (corr = -0.008; accuracy ≈ independence baseline; all-neutral baseline = 0.467 beats the model). Written up in
docs/superpowers/specs/2026-07-01-adaptive-retune-loop-design.md. Fine-tuning spec to follow on a separate branch.Testing
27/27 tests pass (
uv run pytest tests/), including new regression tests for the dedupe bug, archive behavior, and threshold step-down.