feat: SOTA AutoML engine, serve-safe GenAI, and publishable-grade benchmarks#17
Merged
Conversation
added 30 commits
July 4, 2026 16:33
… panel; brier is lower-is-better; NaN primary Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…ansion, target encoding and missing indicators Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…sification labels Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…ning callback and elapsed time in policies Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…stacking modes)
Wire out-of-fold collection (cross_val_predict on the shared seeded splitter) for
the top ensemble_size candidates and dispatch on ensemble mode ("auto"/"greedy"/
"stacking"/True/False). The greedy Caruana vote is measured on OOF and, in "auto",
served only when it beats the best single candidate; its score is added to the
leaderboard as a greedy_weighted_ensemble entry.
Rename _resolve_cv -> _splitter (single shared splitter helper). Make
WeightedVoteEstimator sklearn-typed (estimator-type tags + score) so the now
default-on vote works with permutation-importance explainability and calibration.
Advanced sample requests ensemble="stacking" explicitly (True now maps to "auto").
Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…econds AutoML(time_budget=) allocates an equal share of the remaining wall clock to each not-yet-run candidate (monotonic deadline); the first candidate always runs, later ones past the deadline are skipped with a leaderboard note and never enter selection. LeaderboardEntry gains cv_std/fit_seconds/n_trials/note (appended, defaulted) populated from each candidate's search, and leaderboard_table() renders the enriched columns. Search-side timeout/seed_trials/elapsed_seconds landed in b4c9264; this adds the frozen-result and Optuna-timeout tests from the brief that were still missing. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
AutoML._objective now cross-validates fold-by-fold and, when the search policy injects a report_fold callback, reports the running-mean score after each completed fold. The Optuna policy's callback raises TrialPruned to stop unpromising trials early (MedianPruner, n_startup_trials=5) while keeping candidate failures a -inf-scored, non-aborting outcome. A report_fold that raises is left to propagate: pruning is policy-owned control flow, never swallowed as a candidate failure. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
Add a curated, size-guarded static portfolio (AutoGluon/TabRepo-style) of known-good configs per trainer and wire the AutoML facade to enqueue them as Optuna seed_trials before tuning. The facade drops any seed whose keys fall outside the trainer's declared param space (Optuna's enqueue_trial warns on unknown params). A `portfolio` constructor knob accepts a custom provider; None uses the built-in, and seeds are skipped entirely when not tuning. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
Below a 20% minority ratio, AutoML.fit now injects per-library balanced class weights as estimator defaults that the hyperparameter search can still override: sklearn/LightGBM class_weight='balanced', CatBoost auto_class_weights='Balanced', and XGBoost binary scale_pos_weight = n_negative/n_positive. Gated by AutoML(auto_class_weight=True) and only for classification. The injected defaults are merged under search params and recorded in the leaderboard entry params. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…uard Add TabPFNTrainer (name=tabpfn, family=tabfm): classification-only, zero tuning (empty param_space), CPU classifier. A module-level _tabpfn_weights_cached() gate plus supports_dataset(X, y) keeps TabPFN to its competitive regime (<=10k rows, <=500 features) and never triggers an implicit weight download. Register it in _default_trainers() and the models auto-configuration, both gated on tabpfn importability like the boosting libs. Teach the AutoML facade to honor an optional supports_dataset(X, y) trainer guard, filtering candidates after the supports(task) check and raising a clear FireflyDataScienceError when every trainer declines the dataset. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…ion) With tabpfn installed and its checkpoint cached, TabPFNTrainer joins _default_trainers() and costs ~50-90s per breast_cancer-sized classification fit, blowing the local suite up to ~20+ minutes (the advanced-AutoML sample test alone exceeded 30 minutes: TabPFN refit across CV, stacking-internal CV and calibration). CI (no tabpfn) was unaffected. - tests/_helpers.py (new): fast_trainers() pins the always-available sklearn trio for tests whose subject is not trainer coverage; hide_tabpfn_weights() mirrors CI (no cached checkpoint -> supports_dataset declines) for sample smoke tests that fit default-trainer AutoML inside sample code we cannot pass trainers= into. - Pinned trainers=fast_trainers() in test_calibration, test_ensemble (AutoML sites), test_cv_and_prauc_selection, test_automl::test_classification_end_to_end, test_automl_genai_wiring, explainability::test_automl_result_explains_the_winner_on_real_data. - hide_tabpfn_weights() in the advanced-AutoML/lumen/tutorial sample smoke tests. - pyproject.toml: pytest pythonpath gains "." so tests can import tests._helpers. Tiny iris/synthetic default-AutoML tests are left alone (TabPFN is fast there), and test_default_trainers.py plus the TabPFN tests in tests/models/test_trainers.py keep covering TabPFN's default participation. Full local suite with TabPFN active: 189 passed in 194s. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…rendering HistGradientBoostingTrainer now defaults to early_stopping=True (explicit params still override it). AutoML's fit loop now records each candidate's fold-score std (cv_std), keyed off the winning trial's score, instead of leaving it at its unpopulated 0.0 default. leaderboard_table() renders an unpopulated cv_std as an em dash rather than a misleading "0.0000".
…ning pipeline Accepted LLM feature snippets were train-only, so predict(raw_X)/evaluate(test) broke after feature engineering: the fitted winner expected engineered columns absent from raw serving data. FeatureCodeTransformer (BaseEstimator + TransformerMixin) embeds the accepted snippets as the served pipeline's first step and replays them through the sandboxed FeatureCodeExecutor at predict time. The facade prepends it to whatever _build_winner returns and fits on the RAW pre-engineering matrix, so there is no train/serve skew. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…feedback GenAIFeatureEngineer gains rounds (default 1). Rounds >= 2 send an EngineeringFeedback (contract C8: round_index, current_score, metric, accepted gains, rejection reasons) to the proposer — only if its propose() signature accepts a feedback kwarg (legacy proposers stop after round 1). Proposals are deduplicated by whitespace-normalized code across all rounds; an all-duplicate round ends the loop. AgentFeatureProposer accepts the feedback kwarg and renders it into the LLM prompt. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
The FeatureEngineerPort and GenAIFeatureEngineer.engineer gain optional metric/cv kwargs (contract C8). The facade passes its resolved metric and its resolved splitter (self._splitter(task)) so the gate measures exactly what selection optimizes — this also fixes the previous divergence where the gate cross-validated on unshuffled bare-int folds while selection shuffled. Legacy engineer signatures keep working: the facade signature- sniffs and only passes the kwargs the engineer accepts. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…fer default model to agentic Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
Add LLMSeedProposer (contract C10) in search/llm_seeds.py: an LLM proposes hyperparameter configs to warm-start Bayesian search, each validated against the trainer's declared ParamSpace (unknown keys dropped, numeric clamped, categorical must be a declared choice; invalid/empty/duplicate configs dropped, capped at k). Lazy FireflyAgent like AgentFeatureProposer; model default None defers to the agentic configured default; fully testable LLM-free via an injected agent or pydantic-ai TestModel. Wire the facade (contract C6): AutoML(seed_proposer=None) merges the proposed, already-validated seeds into the portfolio seed_trials before optimize, via a best-effort _llm_seed_trials helper that logs and yields nothing on LLM failure so warm-start never aborts fit. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…l LLM narrative Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
validate_config clamped stepped IntParam values into [low, high] but left them off the step grid, so OptunaSearchPolicy's enqueued warm-start trials were silently resampled by suggest_int(..., step=...) whenever the LLM proposed an off-grid value (e.g. 273 for a step=50 space). Now clamped values are rounded to the nearest step-grid point (skipped for log-scaled IntParams, which have no step) and re-clamped in case rounding overshoots the bound.
AutoMLConfig (cv/n_trials/time_budget/metric/ensemble/ensemble_size/ calibrate/auto_class_weight) joins the root config; the automl_backend bean now builds AutoML from config.automl. metric stays a fit-time argument per contract C6 (documented on the config and the bean). GenAI audit_path/default_model wiring already landed in a prior task. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
Env vars are always strings, so FIREFLY_DATASCIENCE_AUTOML__ENSEMBLE=false
left the literal string "false" on AutoMLConfig.ensemble (pydantic's
bool | str union does not coerce it), which then failed _ensemble_mode's
`is True`/`is False` identity checks and raised
FireflyDataScienceError("Unknown ensemble mode 'false'"). Add a
field_validator(mode="before") that case-insensitively maps
true/1/yes -> True and false/0/no -> False, leaving auto/greedy/stacking
untouched.
…th unit tests Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
… stats replace pooled Wilcoxon Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…m and metered cost Delete the obsolete _apply_accepted hand-replay glue and score both ablations end-to-end through the served pipeline (best_model.predict_proba / evaluate on RAW test frames), which now embeds accepted snippets via FeatureCodeTransformer. Key-gate the proposer: real AgentFeatureProposer when ANTHROPIC_API_KEY/ OPENAI_API_KEY is present, deterministic StaticFeatureProposer fallback for offline runs. Add the OpenML credit-g (id 31) real-data arm over repeated splits, and derive the cost line from agentic metering only (no fabricated strings). Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
added 6 commits
July 4, 2026 23:33
…c plumb-through - AutoMLResult.best_score/report()/tracking key off the SERVED model's leaderboard entry; a losing "auto" greedy ensemble stays listed but is marked note="not served" (its pooled-OOF score can still sort first) - OptunaSearchPolicy.n_trials counts only COMPLETE+PRUNED trials — WAITING enqueued seeds never inflate the count - AutoML gains default_metric (constructor) so automl.metric from config is no longer inert; the automl_backend bean wires it through - log when supports_dataset filters a candidate (mirrors time-budget skips) - report() renders unmeasured cv_std as em dash, same as leaderboard_table() - AutoML.__init__ docstring: time_budget bounds the search loop only - test hygiene: restore scientific_eval.N_TRIALS via monkeypatch; drop the duplicate genai-defaults config test Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…limit TabPFN 2.x raises on CPU fits above 1000 samples; CV folds could pass the old 10k guard while the winner's full-train refit crashed. Align the guard with the library's real limit so TabPFN only enters leaderboards it can serve. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…ate claims, honest sandbox tiers Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…els in beat_baseline Both arms now share one seeded StratifiedKFold(shuffle=True, random_state=0) per dataset, so cross_val_score and AutoML score on identical folds — closing the unshuffled-vs-shuffled mismatch that manufactured a spurious -0.12 on blood-transfusion. Firefly runs ensemble=False so best_score is a single-model fold-mean, commensurate with the baseline fold-mean. Labels are three-way (WIN/tie/LOSS) on the existing 0.0005 band and the summary reports wins/ties/losses truthfully. RESULTS.md head-to-head refreshed from the re-run: wins 6, ties 0, losses 0, mean Δ +0.0344. Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
…ark prior-cycle sample figures Claude-Session: https://claude.ai/code/session_01Q8DCCC3EfYS4CbwGADTfqd
This was referenced Jul 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SOTA AutoML engine + serve-safe GenAI + publishable-grade benchmarks
36 commits across five workstreams, every task TDD'd and independently reviewed. Final state: 267 tests passing, ruff/pyright-clean, and every number in RESULTS.md produced by real runs of the bundled harnesses on this cycle.
Engine (classical AutoML)
search/portfolio.py).ensemble="auto"): measured on OOF and served only when it beats the best single model; losing ensembles are listednot served.time_budget(search-loop wall-clock), automatic class-imbalance weighting, TabPFN in the leaderboard behind asupports_datasetguard aligned with its real 1000-row CPU limit, enriched leaderboard (fold std, fit seconds, honest trial counts).GenAI (governed, serve-safe)
FeatureCodeTransformerstep in the winning pipeline —fiton raw →predicton raw (previously train-only: a production-breaking gap).result.report().Scientific benchmarks (all numbers from real runs)
benchmarks/_stats.py, unit-tested); a docs honesty gate (tests/docs/test_claims.py) permanently bans the corrected overclaims.Infra