Autonomous campaign: MLE-STAR kaggle solver, model bakeoff, engine guards, graded benchmark results - #36
Open
cgpadwick wants to merge 28 commits into
Open
Autonomous campaign: MLE-STAR kaggle solver, model bakeoff, engine guards, graded benchmark results#36cgpadwick wants to merge 28 commits into
cgpadwick wants to merge 28 commits into
Conversation
The artifacts list only covered the ledgers and report, so a cross-box resume restored a workspace with no model.py/train.py/checkpoint — the hillclimb would have to re-derive its own solution mid-loop (this is exactly what made run 1815-ec5b unrecoverable when its box died). It also meant a finished run's submission.csv never reached the mirror, so nothing could be audited or re-graded once the node was gone. Mirror the code files, best checkpoint, submission, eval/grading outputs, and the analysis docs alongside the ledgers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An agent step may name `model: <id>`; hydrate builds a same-type provider for it from the flow's provider spec (one cached instance per model). Cost lever from the kaggle plan: cheap model for critics, strong model for propose/implement, one flow. Rules: CLI --model forces one model everywhere (step ids may not exist in another provider's namespace once --provider changes); an injected provider object (tests) ignores step models. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…(P1) comp_understanding and propose may now web_search for recipes for the task FAMILY — MLE-STAR's single biggest win over prior agents. Guards are mechanical, per the plan's decision #2, not honor-system: - engine: SAAGE_SEARCH_BLOCK_DOMAINS drops results from listed domains (+subdomains) before the model sees them; a backend-synthesized answer is dropped too since it can't be domain-filtered - skills: queries must characterize the task generically — never the competition name/id; every query lands in saage.log for audit - node bootstrap installs saage[search] so ddgs exists on remote boxes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The measurement harness from the plan's P0/M3: one command fans a set of mle-bench competitions out to cloud boxes (spawn + handoff per comp), polls the R2 mirror, and collects each finished run's grade, token cost, and GPU hours into benchmark_journal.jsonl — from which it renders BENCHMARK_RESULTS.md, the medals-per-dollar table. The driver owns no remote logic: actions shell out to the saage remote CLI, observation reads the same R2 mirror saage uses, and it is restartable — a killed sweep is re-collected any time with `bench.py collect --runs <ids>`. Sweeps default to SAAGE_SEARCH_BLOCK_DOMAINS=kaggle.com so retrieval-grounded runs stay uncontaminated. Pure pieces (journal upsert, table render, summary/ checkpoint parsers) are unit-tested offline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three cheap guards from the plan's P4, costing one extra LLM call per RUN (not per iteration): - data_audit: a one-time agent step after the baseline that audits the solution for train/test leakage and lists provided-but-unused data files, appending findings to research_log.md — so all 30 hill-climb iterations see them (unused modalities are the most common reason a solution plateaus below medal range) - proposal_critic: rejects proposals that would introduce leakage (preprocessing fit on all rows, target-derived features, tuning on the reported validation split) - verify_training: fails a too-good-to-be-true score (logloss ~0 / AUC ~1.0) as suspected leakage instead of letting keep_or_revert bank it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autonomous flows (kaggle sweeps, overnight hill-climbs) had loop/step bounds but no spend bound. The agent loop now checks the running cost estimate before each model call and raises CostLimitExceeded once SAAGE_MAX_COST_USD (or `saage run --max-cost`) is crossed. The raise happens between model calls, so the checkpoint holds the last completed step — raise the cap and `saage resume` continues the run. A model with no known rate can't trip the cap; warn once so an unenforceable cap is visible (SAAGE_PRICES adds rates). Docs in README + AGENTS.md; per-step model override documented in AGENTS.md too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kaggle_solver and interactive_demo were missing from the example-flows table; the status line still said ~800 lines / 9 modules from v1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thunder's a6000 (~$0.35/hr) is the cheapest box saage can rent — the kaggle plan's P0 sweep economics named it explicitly. thunder_api.py mirrors lambda_api.py: pick_gpu (auto = cheapest with capacity), wait_running that deletes the instance on boot timeout (never leak a billing node, transient poll errors never abort the wait), and the same Cloudflare UA requirement. API verified against the tnr CLI source (the published OpenAPI spec has empty request schemas) + live read-only probes: create returns the instance's private ssh key exactly once — spawn persists it under ~/.saage/ssh/thunder_<id> before anything that can fail, and registers the target with it. terminate now searches Lambda then Thunder for the target's IP and stops billing in whichever cloud owns it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…art) Validated against a real a6000 spawn/terminate cycle; three facts the docs/CLI don't tell you: - valid vCPU counts are per-GPU and server-side only (a6000: [4 6], tnr's static choices claim [4 8 16 32]) — default to 4 and retry once with the smallest count a vCPU validation error lists - k8s-provisioned instances NAT ssh on metadata 'port' (seen: 31279), not 22 — wait_ssh gains a port param and the target registers with it - cold-start provisioning takes ~10 minutes — wait_running gets a 20min budget at the spawn call site Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… memory (P2/P3/P5) Port of MLE-STAR's load-bearing loop shape (arXiv:2506.15692) into the deterministic flow, at the same 30-experiment budget: - P2 ablation targeting: the flat 30-iteration hillclimb becomes outer_phases(6) × inner_iterations(5). Each phase starts with an ablation agent that writes and runs a cheap component-knockout study, summarizes it, and names TARGET_BLOCK; propose must refine that component (with an explicit escape hatch after 2+ consecutive reverts on it). Targeted refinement replaces 'change something'. - P3-lite self-ensemble: after final training, an ensemble agent builds one strategy (seed ensembling / checkpoint averaging / TTA) and re-evaluates; the existing deterministic keep_or_revert keeps it only if eval_results.json improved — MLE-STAR's candidate-merging without needing parallel runs. - P5 cross-competition memory: bench.py collect distills each graded run (outcome header + its research log) into memory/<comp>.md in the flow dir; a stage_memory command copies notes into the workspace and comp_understanding reads them. Cold starts get warmer every run. Loop knobs (outer_phases, inner_iterations) are shared-store values, so --set can reshape the budget per run. Integration tests cover the new wiring: target_block capture, nested exit reasons, tie-reverted no-op ensemble ledger row. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nal+memory from run fdc6 First real collect: the 2026-07-02 spooky run (val logloss 0.3207, medal unknown — it predates the grade.py fix) lands in the journal and becomes the first cross-competition memory note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….24 LLM / 2.4 GPU-h bench.py collect fixes from first real use: ledger state key is started_at (date + GPU-hours columns now fill), float scores render at 4 sig figs. dc9a is the first run graded end-to-end by the flow itself: test logloss 0.4129, above the human median, no medal — the medals-per-dollar journal is live. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…se for P4 The old-flow nomad run hill-climbed val RMSLE to 0.0518 (silver-range) but graded 0.1424 on test — a 2.7× generalization gap on a ~2400-row dataset, below median. That failure mode (validation overfit / protocol leakage) is precisely what the new flow's data_audit step and leakage critics were added to catch; the memory note now carries the gap so future nomad runs start warned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same competition, model, and GPU: the MLE-STAR flow graded test logloss 0.3481 (above median) vs the flat hillclimb's 0.4129. Val→test gap near zero (0.3526→0.3481). The self-ensemble stage demonstrably paid: the 100-epoch final train overfit to 0.3901 and the agent's 3-seed ensemble recovered 0.3526, kept by the deterministic gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- flow default model: deepseek-v4-pro; the seven mechanical-judgment steps (critics, verifiers, summarize) override to v4-flash via per-step model: — the E3 cost lever in its intended shape (~5x cheaper where quality doesn't move the score) - build_baseline: stop prescribing the validation protocol (the 80/20 example read as an instruction — and single-split selection overfit is exactly what sank the nomad run). The agent owns the protocol; the contract only demands the reported score honestly estimate leaderboard performance. The nomad memory note carries the 2.7x gap lesson for future runs to reason from. - pricing: exact v4-pro/v4-flash rates so --max-cost and run summaries stay grounded Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Longest-key-wins is the designed behavior; the old assertions used v4-flash as an example of the generic family match, which the new exact keys correctly outrank. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the flow Passes through to `saage run --model` on the node (forces one model for every step, same semantics as local). Recorded in the manifest so a cross-box resume keeps it. The tool for model experiments: same flow, same comp, different brain per arm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ified) Both ran the guessing_game tool-use flow cleanly via OpenRouter — qualified as kaggle-solver brains. Rates from the OpenRouter list so --max-cost and run summaries stay grounded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`saage remote fetch --workspace` can land multi-GB checkpoints under results/ inside the repo; the engine rsync then ships them to every new node and times out (killed both gold-run handoffs tonight). The node needs the engine source — never local run results, contrib flows, or egg-info. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… (10x) val 0.0518 -> test 0.5154. Two nomad runs, different flows and different brains, both destroyed by validation leakage on a 2,400-row dataset — the one-shot data_audit and prompt-side critics don't cut it there. Next engineering priority named in the README: MLE-STAR's per-solution code-level leakage check before every execution. Model-bakeoff datapoint: minimax-m3 drove the full 40-experiment MLE-STAR loop competently (real domain feature engineering, clean tool use, $19.77 LLM) — the failure was protocol, not agentic capability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…edy search MLE-STAR seeds refinement from the best of several independent initial solutions; we seeded from the first one that compiled — and two live runs proved the cost (v4-pro locked into a weak classical pipeline twice; greedy hill-climbing can't escape its starting basin). The baseline stage is now a counting_loop over baseline_candidates (default 2): build -> smoke -> short-train -> keep_or_revert with the new --baseline auto mode (first candidate records, later ones must beat it — nan-safe reset between candidates). build_baseline must produce a genuinely different family when the log shows an earlier candidate; the winner's code survives in git and seeds the hill-climb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…emble next 1ee2 (A100, flash, 40 experiments): val 0.3524 vs 759b's 0.3526 — a dead heat — but test 0.3638 vs 0.3481. Deepseek-class single solutions plateau at ~0.35 on spooky; the medal path is cross-run ensembling and stronger families, not more iterations. $1.85 of LLM for the whole 42-experiment run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cross-family ensembling Equal-weight geometric blend of the two strong spooky submissions graded 0.29470 vs 0.34808/0.36381 alone — 15% from diversity, 0.0009 short of the bronze threshold. Recipe was fixed a priori (blend.py: equal weights, declared combiner; no test-tuning), singles' local grades match their run-time grades exactly. A blend of runs isn't a legal benchmark submission, so the ensemble skill now surfaces the in-run equivalent: kept solutions and losing baseline candidates survive in git history — resurrect a genuinely different family at reduced budget and blend cross-family (geometric for logloss). Diverse errors cancel; same-family seeds barely do. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… run Seen live: a hill-climb experiment's train.py silently fell back to CPU (GPU 0%, 2h of CPU time) and would have held the flow for hours; command steps had no bound (agents have max_steps, loops have max_iterations — commands had nothing). `timeout:` on a command step kills the process at the bound and returns exit 124 with a timeout note in stderr, so retry_loops/captures treat it as a failed attempt (nan -> revert) and the run continues. Kaggle flow caps train commands (2h short / 5h final). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sm fired ec27 (multi-seed baselines + cross-family ensemble, flash, A10): val 0.3496 (campaign best), test 0.3581, above median. The seeding loop caught a broken 9.36-logloss baseline candidate live; the ensemble agent resurrected the losing family from git history and built an LGBM branch, which the keep-gate correctly reverted (too weak on text). Offline 3-way blend of the strong runs graded 0.3013 — worse than the 2-way 0.2947: diversity beats strength-in-numbers, and blend probing stops here (further pair-shopping would be test-set fitting). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR upgrades saage’s flagship kaggle_solver flow to an MLE-STAR-shaped autonomous campaign with measured benchmark outputs, while adding engine/runtime guards (per-step model selection, command timeouts, cost cap, search domain blocklist) and expanding remote provisioning (Thunder Compute) to support the campaign’s GPU automation.
Changes:
- Add engine controls: per-step
model:overrides for agent steps, command-steptimeout:, and a spend cap (--max-cost/SAAGE_MAX_COST_USD). - Add retrieval contamination guard:
SAAGE_SEARCH_BLOCK_DOMAINSto mechanically filterweb_searchresults by domain/subdomain. - Expand Kaggle solver + benchmarking + remote: MLE-STAR outer/inner loop structure, baseline seeding, data audit, self-ensemble stage, benchmark sweep/journal tooling, and Thunder Compute spawn/terminate support.
Reviewed changes
Copilot reviewed 42 out of 42 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
tests/test_step_model.py |
Adds coverage for per-step agent model: overrides and provider caching/locking behavior. |
tests/test_search.py |
Adds tests for domain blocklisting and end-to-end web_search filtering. |
tests/test_pricing.py |
Updates pricing/rate matching expectations and override behavior. |
tests/test_kaggle_keep_or_revert.py |
Adds tests for --baseline auto semantics. |
tests/test_command_action.py |
Adds test coverage for command timeouts returning exit 124. |
tests/test_bench.py |
Adds offline unit tests for bench.py parsers/journal/table/memory-note helpers. |
tests/test_agent.py |
Adds tests for the cost cap / “money fuse” behavior. |
tests/remote/test_thunder_api.py |
Adds unit tests for Thunder provisioning helpers (gpu picking, wait, vCPU retry). |
tests/remote/test_scripts.py |
Adds test for remote start script --model passthrough. |
tests/integration/test_kaggle_solver.py |
Updates integration coverage for new Kaggle solver steps (audit/ablation/ensemble/baseline loop). |
saage/search.py |
Implements SAAGE_SEARCH_BLOCK_DOMAINS filtering in web_search. |
saage/remote/thunder_api.py |
Introduces Thunder Compute API client + provisioning helpers. |
saage/remote/scripts.py |
Adds remote --model passthrough and installs saage[search] on nodes. |
saage/remote/resume.py |
Persists/restores --model in remote resume run specs. |
saage/remote/lambda_api.py |
Extends wait_ssh to support non-22 SSH ports (NAT’d SSH). |
saage/remote/handoff.py |
Records model override in run manifest; expands engine exclude patterns for remote packaging. |
saage/remote/cli.py |
Adds Thunder support for spawn/terminate and remote handoff --model. |
saage/pricing.py |
Adds/updates model pricing keys for mid-2026 models and preserves longest-substring matching. |
saage/nodes.py |
Adds command-step timeouts and converts timeouts into failed step outputs (exit 124). |
saage/hydrate.py |
Adds per-step provider selection and provider caching keyed by model id. |
saage/cli.py |
Adds --max-cost CLI flag to set SAAGE_MAX_COST_USD for the run. |
saage/agent.py |
Enforces cost cap between model calls; adds CostLimitExceeded. |
README.md |
Documents cost cap and remote provisioning updates; expands flow catalog notes. |
flows/kaggle_solver/verify_training/skill.md |
Strengthens critic guidance to fail on suspiciously too-good validation (leakage signal). |
flows/kaggle_solver/README.md |
Replaces placeholder results with benchmark table + methodology notes, including contamination guards. |
flows/kaggle_solver/propose/skill.md |
Adds ablation-targeted refinement requirement and allows limited web search usage. |
flows/kaggle_solver/proposal_critic/skill.md |
Adds explicit leakage-risk rejection criteria. |
flows/kaggle_solver/memory/spooky-author-identification.md |
Adds cross-run memory artifact for a graded run. |
flows/kaggle_solver/memory/nomad2018-predict-transparent-conductors.md |
Adds cross-run memory artifact for a graded run. |
flows/kaggle_solver/keep_or_revert.py |
Adds --baseline auto mode for multi-candidate baseline seeding. |
flows/kaggle_solver/flow.yaml |
Reshapes the flow to MLE-STAR phases + baseline seeding + audit + ensemble + new artifacts. |
flows/kaggle_solver/ensemble/skill.md |
Adds post-training ensemble stage instructions with deterministic keep gate. |
flows/kaggle_solver/data_audit/skill.md |
Adds one-time leakage/data-usage audit step that appends to research_log.md. |
flows/kaggle_solver/comp_understanding/skill.md |
Enables web-search-grounded SOTA summary with benchmark-integrity rules + memory ingestion. |
flows/kaggle_solver/build_baseline/skill.md |
Enforces baseline candidate diversity and clarifies validation protocol requirements. |
flows/kaggle_solver/blend.py |
Adds offline cross-run blending tool (geometric/arithmetic mean). |
flows/kaggle_solver/BENCHMARK_RESULTS.md |
Adds generated benchmark results table. |
flows/kaggle_solver/benchmark_journal.jsonl |
Adds benchmark journal source-of-truth records. |
flows/kaggle_solver/bench.py |
Adds sweep/collect/table tooling and cross-run memory extraction. |
flows/kaggle_solver/ablation/skill.md |
Adds ablation analyst step that produces TARGET_BLOCK for refinement targeting. |
docs/kaggle_solver_plan.md |
Updates plan status and progress notes through P5. |
AGENTS.md |
Documents per-step model:, command timeout:, domain blocklist, and --max-cost. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+138
to
+141
| kept = [r for r in results | ||
| if not any(_host(r.url) == d or _host(r.url).endswith("." + d) | ||
| for d in blocked)] | ||
| return kept, len(results) - len(kept) |
Owner
Author
There was a problem hiding this comment.
Fixed in a9de517 — host is now parsed once per result (allowed() closure).
Comment on lines
+199
to
+207
| log.warning(" ✗ %s → timed out after %.0fs (killed)", self.id, | ||
| self.timeout or 0) | ||
| out = (e.stdout or b"").decode("utf-8", "replace") \ | ||
| if isinstance(e.stdout, bytes) else (e.stdout or "") | ||
| err = (e.stderr or b"").decode("utf-8", "replace") \ | ||
| if isinstance(e.stderr, bytes) else (e.stderr or "") | ||
| return {"exit": 124, "stdout": out, | ||
| "stderr": err + f"\nERROR: command timed out after " | ||
| f"{self.timeout:.0f}s and was killed"} |
Owner
Author
There was a problem hiding this comment.
Fixed in a9de517 — %g in both the log line and the stderr note, so sub-second timeouts report their real value.
…out message keeps precision - apply_blocklist re-parsed the URL host for every blocked domain; compute it once per result - the command-timeout log/error rounded to whole seconds (%.0f), which reports a 0.3s timeout as 0s — %g keeps the real value Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cgpadwick
commented
Jul 8, 2026
| @@ -0,0 +1,93 @@ | |||
| # nomad2018-predict-transparent-conductors — run kaggle_solver-20260704-0617-b546 | |||
|
|
|||
Owner
Author
There was a problem hiding this comment.
hmmmm, why is the memory file from a specific run getting checked in?
Owner
Author
There was a problem hiding this comment.
You're right — this is run output, not flow source. Fixed in 1dbeb1d: memory/ is gitignored + untracked. It still lives in the flow dir ON DISK (handoff rsyncs the working tree, so stage_memory and remote runs keep working — committing was never actually required for that; I'd conflated the two).
| @@ -0,0 +1,112 @@ | |||
| # spooky-author-identification — run kaggle_solver-20260704-2106-ec27 | |||
|
|
|||
Owner
Author
There was a problem hiding this comment.
You're right — this is run output, not flow source. Fixed in 1dbeb1d: memory/ is gitignored + untracked. It still lives in the flow dir ON DISK (handoff rsyncs the working tree, so stage_memory and remote runs keep working — committing was never actually required for that; I'd conflated the two).
| @@ -0,0 +1,7 @@ | |||
| {"above_median": "?", "competition": "spooky-author-identification", "date": "2026-07-02", "gpu_hours": "10.2", "llm_cost_usd": "4.9468", "medal": "unknown", "model": "openrouter", "run_id": "kaggle_solver-20260702-0425-fdc6", "test_score": "?", "tokens": "14,363,807", "val_score": "0.3207"} | |||
Owner
Author
There was a problem hiding this comment.
I view this is a specific experiment output...why is it checked into a general flow?
Owner
Author
There was a problem hiding this comment.
Agreed — experiment output. Fixed in 1dbeb1d: gitignored + untracked; bench.py keeps writing it locally. The curated table in the flow README is the only committed record.
Owner
Author
There was a problem hiding this comment.
Agreed — experiment output. Fixed in 1dbeb1d: gitignored + untracked; bench.py keeps writing it locally. The curated table in the flow README is the only committed record.
| @@ -0,0 +1,15 @@ | |||
| # kaggle_solver benchmark results | |||
|
|
|||
| Autonomous runs of `flows/kaggle_solver` graded with `mlebench grade`. | |||
Owner
Author
There was a problem hiding this comment.
Same fix, 1dbeb1d: it's generated from the journal by bench.py table, so it's gitignored now; README links point at the curated results table instead.
…ts table Review feedback on #36: these are experiment outputs, not flow source. They stay in the flow dir ON DISK (handoff ships the working tree, so stage_memory and remote runs still see them) but are gitignored and untracked. The curated results table in the flow README remains the committed record; links to the generated BENCHMARK_RESULTS.md now point at it instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Three days of autonomous work (2026-07-03 → 07-05): the kaggle solver rebuilt in MLE-STAR's shape and validated by live A/B, plus the engine/remote features the campaign needed along the way. 30 commits, suite 396 → 431 tests, all offline.
Kaggle solver (the flagship)
SAAGE_SEARCH_BLOCK_DOMAINS— engine drops kaggle.com results; queries logged for audit), one-time leakage/data-usage audit, self-ensemble stage with a deterministic keep-gate, cross-competition memory (bench.py collect→memory/<comp>.md→stage_memory), multi-candidate baseline seeding (--baseline auto— a bad seed dooms greedy search; caught a broken 9.36-logloss candidate live).bench.pysweep driver +benchmark_journal.jsonl→BENCHMARK_RESULTS.md;blend.pyfor pre-registered cross-run blending.Engine
model:on agent steps (cheap critics, strong proposers) — CLI--modelstill forces globally--max-costmoney fuse (stops between model calls; run stays resumable)timeout:(a hung train fails the step, never the run — born from a live CPU-fallback hang)Remote
spawn --provider thunder, REST API, live-validated: per-GPU vCPU validation, NAT'd ssh ports, 10-min cold starts); cross-cloudterminatehandoff --modelpass-through (same flow, different brain per arm)ENGINE_EXCLUDEScoversresults/(multi-GB fetched checkpoints no longer ship to nodes)Model-bakeoff findings (live GPU runs)
Full narrative + charts: campaign report artifact (linked in session).
🤖 Generated with Claude Code