diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 163e92e..db6bd7d 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1135,6 +1135,49 @@ game** (simpler than the glob-exclusion a shared `nn.pg` needs); the `.ckpt` par polynomial approximations) only if uniform single-source is a hard requirement, gated on an argmax-parity spike (MC0). Plan MC0–MC5. **Payoff:** the last two per-viewer AI sockets → zero server inference. +## M34 — Snake: look-ahead search (client-side, single-source) *(branch `m34-snake-search`, off `master`; see `SNAKE_SEARCH_PRD.md`)* ⏳ + +**Problem.** Shipped client-side Snake (M33) plays masked-greedy **one-step** over the 177-dim net and is stuck at the +**~50 food@12** reactive plateau M27 already charted (capacity/features/reward/horizon all swept → ~50; a reactive net +can't avoid a trap that forms several moves ahead). The reachable-free-space input + the 1-ply flood-fill shield M27 +added are *already shipped* — the missing lever is **planning more than one ply**, not more training. + +**Fix.** Port PR #11's proven idea — net-guided multi-ply look-ahead (food@12 ~50 → ~78.6) — into the **single-source +`snake_solver.pg`**. (PR #11 itself is unmergeable: it predates M32/M33's Polyglot + client-side rewrite — server-side C# +`SnakeSearchAgent` + 39-dim ray obs; `CONFLICTING`.) `chooseActionSearch` runs a receding-horizon **beam search** over +cloned envs with **pure-survival leaf scoring** (board-full win ≫ everything; a death *delayed* beats a death now; else +`food·w − trapPenalty·[reachable **Design decision (2026-07-10).** The strength comes from an exact **flood-fill survivability search** (reuses the +> `reachableFreeSpace` the net already carries as an input). The **biggest single lever** turned out to be an +> **anti-fragmentation term** — scoring the *fraction of currently-free cells still reachable* (the reachability-ratio +> idea originally floated as a net input, but far more effective in the **search leaf score**): it lifts food@12 ~71 → ~81 +> (+14%, robust across seed bases). A per-node net evaluation, by contrast, added **no** strength for ~9× the latency, so +> the trained net is kept in the loop only as a cheap **tie-breaker between equally-safe moves**. This reframes "make the +> Snake net strong": the net's reactive ceiling is real and low, so the *agent* is made strong by search while the *net* +> stays a leaf/tiebreak evaluator. The residual cap (~81 mean; single games hit ~106) is self-traps beyond the horizon — +> left to a future tail-reachability / Hamiltonian endgame. diff --git a/docs/prd/SNAKE_SEARCH_PRD.md b/docs/prd/SNAKE_SEARCH_PRD.md new file mode 100644 index 0000000..d8f39cf --- /dev/null +++ b/docs/prd/SNAKE_SEARCH_PRD.md @@ -0,0 +1,138 @@ +# Snake — net-guided look-ahead search (client-side) — PRD + +**Status:** in progress · 2026-07-10 · branch `m34-snake-search` (off `master`) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M34 · **Depends on:** M33 (Snake single-source `.pg` + fully client-side AI) + +## 1. Problem + +The shipped Snake demo (M33) plays with a **masked-greedy one-step** policy over the trained 177-dim dueling-Q net. +That policy is stuck at **~50 food on 12×12** — and PLAN M27 already established this is a **structural ceiling of a +reactive learned policy, not a training shortfall**: capacity (128→256), features, reward shaping and horizon were all +swept and all plateaued at ~50. A reactive net cannot avoid walking into a region it can no longer fit its body into, +because that trap forms several moves ahead of the one-step decision. + +The observation *already* carries the anti-trap signal M27 added — per-move `reachableFreeSpace / cells` (4 inputs) and +a 1-ply flood-fill **shield** in the action mask — so "add a reachability input" is largely already done. The missing +piece is **planning more than one ply ahead** with that signal. + +## 2. Goal & success criteria + +Make the demo Snake **strong** while keeping it the trained model playing (the net stays in the decision) and fully +client-side (zero server inference, single-source `.pg`). + +- **Gate:** food@12 (12×12, ≥ 20-ep mean) **markedly past the ~50 reactive plateau** — measured **≈ 81** (per-episode + variance is high, ~55–108, since one bad trap ends a game; single games now reach a near-full board). Stretch: a clean 100. +- **No retraining, no observation change** — the existing shipped `snake-net.ckpt` is reused verbatim. +- **Single-source:** the search lives once in `snake_solver.pg` and transpiles to C# (training/eval) **and** the browser + TS director, byte-identically. +- **Browser cadence stays watchable** — one AI move per visible tick; the planner must fit that budget. + +## 3. Key decision — port the idea, don't merge the branch + +PR **#11** (`snake-ray-obs`) already proved the lever: **net-guided multi-ply look-ahead → food@12 ~50 → ~78.6** (3.7× +the original 21). But PR #11 **predates the M32/M33 Polyglot + client-side rewrite** — it is a hand-written C# +`SnakeSearchAgent` driving a **server-side** `SnakeController.Live`, plus a 39-dim ray-cast observation. It shows +`CONFLICTING` against `master` and cannot be merged or cleanly cherry-picked (it resurrects deleted server files and a +different observation). + +**Decision:** take PR #11 as *inspiration* and **re-implement the search inside the single-source `snake_solver.pg`**, +on top of master's current 177-dim net and the `reachableFreeSpace` flood-fill it already ships. The net is kept as the +**leaf evaluator** (PR #11's finding: the survival term carries the search, the net is a marginal tiebreak — but keeping +it in the loop keeps the demo a genuine "watch the trained model" showcase). + +## 4. Design (`snake_solver.pg` → C# + TS) + +`chooseActionSearch(net, maxDepth, beamWidth, foodWeight, trapPenalty, netWeight, spaceWeight, foodDistWeight)`: + +- **Receding-horizon beam search.** From the live state, simulate every legal (non-reversal) line to `maxDepth` plies + on **cloned** envs, keeping the best `beamWidth` survivors per ply; play the first move of the best-scoring line and + re-plan next tick. Snake is deterministic between food, so the look-ahead is exact. +- **Leaf score:** a board-full win dominates; a death is dominated but ranked to prefer a *later* death; a survived leaf = + `foodGained·FoodWeight − TrapPenalty·[reachable < length] + freeSpaceAhead·SpaceWeight + + SpaceRatioWeight·(freeSpaceAhead / freeCells) − headFoodDist·FoodDistWeight`. `freeSpaceAhead` is the max + `reachableFreeSpace` over the 4 next-head cells. The **`SpaceRatioWeight` term is the key addition** — the *fraction* of + currently-free cells still reachable, which penalizes fragmentation the absolute `reachable < length` test misses (the + snake cutting itself off from most of the board while its body still fits the pocket). The trained net enters only as a + **root-move tiebreak** (`maxQ·NetWeight`, one forward per move — see below), not per leaf. +- **Reuses what M33 already shipped:** `reachableFreeSpace` (= PR #11's `FreeSpaceAhead`), deterministic dynamics, the + 177-dim observation, `PgSnakeNet`. +- **RNG-free simulation:** when a simulated line eats, food is respawned at the **first free cell** (`simSpawnFood`), not + a random one — the single source keeps RNG with the caller, so the search must be deterministic to stay byte-identical + across C#/TS. The fictional food cell barely matters: survival scoring drives the search and every tick re-plans. +- **Live env runs with `safeMask: false`** — the planner's multi-ply survival scoring supersedes the reactive 1-ply + shield, and the returned move is always a non-reversal (hence always legal). + +**Public surface.** The internal transpiled `PgSnakeNet` is exposed only through the `SnakeEnv` facade: +`LoadSearchNet(Stream)` (a C# twin of the browser's `snake-net.ts` parser — same RLNC/`dueling-q` bytes) + +`ChooseActionSearch(SnakeSearchConfig)`. + +## 5. Results (measured — C#, shipped 177-dim net, **no retraining**) + +food@12 on 12×12 (mean, high variance — min/max in parens); greedy baseline ≈ 50 (M27). d12/b16, net-tiebreak 50. + +**The anti-fragmentation term (`SpaceRatioWeight`) is the biggest lever** — a paired sweep (same seeds), confirmed on a +second seed base: + +| `SpaceRatioWeight` | food@12 (seed 1, 20 eps) | food@12 (seed 100, 30 eps) | note | +|---|---|---|---| +| 0 (survival only) | 70.3 | 72.6 | prior shipped baseline | +| 50k | 75.8 | — | | +| **100k (SHIPPED)** | **81.3** (max 108) | **80.6** (max 106) | robust peak; ~+10 food (+14%) | +| 200k | 82.2 | 79.2 | ~tied with 100k but nearer the cliff | +| 400k | 76.0 | — | over-weights connectivity → under-eats | + +Other levers (all d12/b16, at `SpaceRatioWeight` 0 unless noted): + +| config | food@12 | latency | verdict | +|---|---|---|---| +| **SHIPPED — d12/b16, net 50, ratio 100k** | **≈ 81** | ~11 ms/move | anti-fragmentation term + survival search + net near-tie breaker | +| net-tiebreak, net 500 (ratio 0) | 67.8 | 10.6 ms/move | a heavy net weight overrides survival moves → slightly hurts | +| net-guided *per node*, net 500 | 74.0 | **89 ms/move** | rejected: ~9× cost, no strength gain (survival carries it) | +| d20/b32 (ratio 0) | 66.2 | 38 ms/move | rejected: deeper/wider MISRANKS under beam pruning | +| _(reference)_ PR #11 net-guided d20/b32 | ~78.6 | server-side | the original pre-Polyglot result (39-dim ray obs) | + +All configs run the **identical** `.pg` search, so the browser reproduces them byte-for-byte. **Findings:** (1) the +fraction-of-free-cells-reachable term (the user's original reachability-ratio idea, used in the *search score* — not as +a net input) is the strongest single lever, +14%; (2) depth has a sweet spot (~12) — deeper misranks; (3) evaluating the +net at every node buys no strength for ~9× the latency, so the net is a cheap root-move tiebreak. The search — not the +net — is the strength lever. + +## 6. Client integration + +`snake-director.ts`: swap the greedy `chooseAction(net)` for `chooseActionSearch(net, …)`, drive the live core with +`safeMask: false`, and tune `SEARCH_DEPTH`/`SEARCH_BEAM` for a watchable move cadence. The shipped `snake-net.ckpt` and +`snake-net.ts` parser are unchanged; `snake_solver.ts` is regenerated from the `.pg` (never hand-edited). + +## 7. Non-goals + +- **Retraining / a new net / a new observation** — out of scope; the reactive net cannot beat ~50 regardless (M27), and + the search delivers the strength on top of the existing net. +- **A guaranteed clean 100** — see §8. + +## 8. Honest ceiling & follow-up + +The residual cap (~81 mean, though single games now hit 106–108 — a near-full board) is self-traps that form *beyond* +the search horizon. A reliable clean **100+** needs a **tail-reachability survival invariant** (guarantee the head can +always reach its own tail → follow it indefinitely) and/or explicit **Hamiltonian-style endgame** play. Proposed as the +next milestone, not built here. + +## 9. Transpiler findings (MintPlayer.Polyglot 0.3.1) + +Two issues surfaced; both were worked around from the `.pg` side. + +- **TS `any`-inference (shaped the search).** The transpiler drops type annotations on local `List` variables (emits + `let x = []`). A first attempt used **parallel `List` frontiers** (`List` + `List` + `List`); + because `childFirst` derived from `beamFirst[n]` and `beamFirst` was reassigned from a list holding `childFirst`, + TS strict-mode saw a **circular `any`** and failed to compile (`TS7022`/`TS7034`). Fix: use a **`record + PgSnakeBeamNode`** as the frontier element (like FruitCake's `PgPlyResult`) — the typed list element breaks the + cycle. Also compute the net's `rootQ` unconditionally (typed by `net.forward`) rather than in a branch. +- **Incremental-rebuild prelude collision (build flake, not shipping).** In a multi-`.pg` project, an *incremental* + rebuild after editing a `.pg` intermittently emits one unit in "standalone" mode (duplicate global + `Option/Some/None` prelude + a **non-`partial`** `PolyglotProgram`) → `CS0260`/`CS0101`/`CS8863`. Root cause: the CLI + doesn't clean its `--out` dir, so a stale `__polyglot_prelude.cs` collides on re-transpile; the target's incremental + gate then caches the bad `.cs`. **Clean/CI builds are unaffected** (verified). Dev workaround: + `rm obj/*/net10.0/polyglot/*.cs` then rebuild. (This is independent of the record above — records build fine cleanly, + as FruitCake proves.) + +Full repro + fix ideas: `docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`. diff --git a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md new file mode 100644 index 0000000..0ac6468 --- /dev/null +++ b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md @@ -0,0 +1,63 @@ +# Polyglot bug handoff — incremental re-transpile emits a duplicate prelude / non-`partial` PolyglotProgram + +**Found:** 2026-07-10, during Snake M34 (adding look-ahead search to `snake_solver.pg`). +**Polyglot:** `MintPlayer.Polyglot.MSBuild` **0.3.1** (transpiler `tools/win-x64/polyglot.exe` 0.3.1). +**Polyglot source:** `C:\Repos\MintPlayer.Polyglot` +**Severity:** breaks *incremental* dev builds after editing a `.pg` in a **multi-`.pg` project**; a **clean build always +succeeds**, so CI / fresh clones are unaffected — but it's confusing and easy to misdiagnose. + +## Symptom + +In a project with **2+ `.pg` files** built by the glob-all MSBuild target (here: `fruitcake_solver.pg`, +`mountaincar_solver.pg`, `snake_solver.pg`), after **editing one `.pg` and rebuilding**, the C# transpile of one solver +unit is intermittently emitted in **"standalone" mode** — it inlines the `Option`/`Some`/`None` prelude (which also lives +in the shared `__polyglot_prelude.cs`) and declares a **non-`partial`** `static class PolyglotProgram` (the other units +emit `static partial class PolyglotProgram` and reference the shared prelude). The C# compile then fails: + +``` +snake_solver.cs(NNN): error CS0260: Missing partial modifier on declaration of type 'PolyglotProgram'; + another partial declaration of this type exists +__polyglot_prelude.cs(3): error CS0101: '' already contains a definition for 'Option' (Some, None) +__polyglot_prelude.cs(4): error CS8863: Only a single partial type declaration may have a parameter list +``` + +## Root cause (what I actually pinned down) + +It is **not** about the source `.pg` content. I initially suspected top-level `record` declarations, but ruled that out: +with the records removed, an *incremental* rebuild still failed, while *clean* builds pass. The differentiator is +**clean vs incremental**, driven by **stale files in the output directory**: + +- The MSBuild target (`build/MintPlayer.Polyglot.MSBuild.targets`, target `PolyglotTranspile`) transpiles the **whole + `.pg` set in ONE CLI invocation** into `$(IntermediateOutputPath)polyglot\`, and the CLI **does not clean that + directory first**. A 2+-`.pg` project also emits a source-less `__polyglot_prelude.cs` there. +- On re-transpile, a **pre-existing `__polyglot_prelude.cs`** (from the previous build) is present in `--out`, and the + CLI emits one solver unit in standalone mode + re-emits the prelude → the duplicate/`partial` collision above. +- The target's incremental gate (`Inputs=@(PolyglotFile);$(PolyglotTool)`, + `Outputs=…%(Filename).cs`) makes it **worse**: once a bad `snake_solver.cs` is written, its mtime is newer than the + `.pg`, so the next build **skips the transpile and recompiles the bad file** — the failure sticks until the generated + `.cs` is deleted. + +### Reproduce +1. Clean-build a 2+-`.pg` project → succeeds; `obj//net10.0/polyglot/` has `__polyglot_prelude.cs` + one `.cs` per + `.pg`, all `static partial class PolyglotProgram`. +2. Edit any one `.pg`; rebuild → intermittently one unit comes out standalone → `CS0260`/`CS0101`/`CS8863`. +3. `rm obj/**/polyglot/*.cs` and rebuild → succeeds again. (Verified: 3/3 fresh builds correct & deterministic.) + +## Fix ideas (for the Polyglot maintainer) + +- **Clean the `--out` directory** (or at least overwrite/rewrite `__polyglot_prelude.cs` deterministically) at the start + of each `build` invocation, so a stale prelude can't flip a unit to standalone. +- Make the standalone-vs-library decision **deterministic and project-scoped**: in a multi-file invocation, *never* emit a + unit standalone — always shared prelude + all-`partial` `PolyglotProgram`. +- MSBuild side: add `__polyglot_prelude.cs` to the target's `Outputs`, or make `PolyglotTranspile` delete prior generated + `.cs` before running, so the incremental gate can't cache a bad transpile. + +## Impact on this repo + +None at ship time — clean/CI builds are correct. During iterative `.pg` editing, if you hit the errors above: +`rm src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs` then rebuild. + +Note: the M34 search *does* use a top-level `record PgSnakeBeamNode` (a typed frontier element — required to avoid a +separate TS `any`-inference failure; see `SNAKE_SEARCH_PRD.md` §9). Records build fine on a clean build (FruitCake uses +them too); this incremental-staleness bug is unrelated to records. If you hit the errors above during iterative `.pg` +editing: `rm src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs` then rebuild. diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs index 2a53f8d..6ebd011 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs @@ -98,6 +98,28 @@ public StepResult Step(int action) public float[] CurrentObservation() => Observation(); + private PgSnakeNet? _searchNet; + + /// Loads the trained dueling-Q checkpoint the look-ahead planner uses as its leaf evaluator — the same + /// RLNC/"dueling-q" bytes the browser's snake-net.ts parses. Call once before . + public void LoadSearchNet(Stream checkpoint) => _searchNet = SnakeNetIo.Parse(checkpoint); + + /// + /// Net-guided multi-ply look-ahead move for the current state (M34): simulates every legal line to + /// plies on cloned envs and plays the first move of the best-scoring + /// one. The planner supersedes the reactive 1-ply shield, so construct this env with safeMask: false — + /// the returned move is always a non-reversal, hence always legal under the reversal-only mask. + /// + /// No net loaded — call first. + public int ChooseActionSearch(SnakeSearchConfig config) + { + if (_searchNet is null) + throw new InvalidOperationException("Call LoadSearchNet(...) before ChooseActionSearch(...)."); + return _core.chooseActionSearch( + _searchNet, config.MaxDepth, config.BeamWidth, config.FoodWeight, config.TrapPenalty, + config.NetWeight, config.SpaceWeight, config.FoodDistWeight, config.SpaceRatioWeight); + } + private float[] Observation() { var core = _core.buildObservation(); diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeSearch.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeSearch.cs new file mode 100644 index 0000000..b66f3d4 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeSearch.cs @@ -0,0 +1,92 @@ +using System.Text; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Snake; + +/// +/// Tuning for the look-ahead planner (, M34). Defaults are the measured sweet +/// spot: depth 12 / beam 16 lifts food@12 from the reactive ~50 plateau to ~70 (deeper/wider MISRANKS under beam +/// pruning and scores worse — measured d20/b32 ≈ 66). The flood-fill survival search does the work; the net only breaks +/// ties between equally-safe root moves, so is deliberately small — a big weight lets the +/// net override a better survival move and slightly hurts (measured net 500 ≈ 68 vs pure-survival ≈ 71 over 12 eps). The +/// algorithm lives once in snake_solver.pg (chooseActionSearch); this record is just the knobs. +/// +/// Plies to look ahead. The sweet spot is ~12; deeper misranks under beam pruning. +/// Live nodes carried to the next ply, best-scoring first. Wider keeps greedy food-grabs that trap later. +/// Reward per food eaten along a line — the dominant term; eating safely beats any non-eating line. +/// Penalty for a leaf whose reachable space can no longer hold the body (a guaranteed future self-trap). +/// Weight on the trained net's Q, applied ONCE per move as a root-move tiebreak (not per node). Small = pure tiebreak; 0 = ignore the net. +/// Weight on reachable free space (absolute) — keep more room open. +/// Pull toward food when no line eats within the horizon (tiebreak by L1 head→food distance). +/// Weight on the FRACTION of currently-free cells still reachable (anti-fragmentation). +/// The biggest single lever found: sweeping it lifted food@12 ~71 → ~81 (+14%, robust across seed bases), peaking +/// ~100k–200k (400k over-weights connectivity and under-eats → back to ~76). 100k is the robust default. +public sealed record SnakeSearchConfig( + int MaxDepth = 12, + int BeamWidth = 16, + double FoodWeight = 10_000, + double TrapPenalty = 50_000, + double NetWeight = 50, + double SpaceWeight = 50, + double FoodDistWeight = 1, + double SpaceRatioWeight = 100_000); + +/// +/// Reads a trained dueling-Q checkpoint (RLNC / kind "dueling-q") into the single-source transpiled +/// the planner evaluates leaves with. This is the C# twin of the browser's +/// snake-net.ts parseSnakeNet — same byte layout, so C# eval and the browser score identical positions. +/// Kept internal: is an internal transpiled type, exposed only through . +/// +internal static class SnakeNetIo +{ + private const uint Magic = 0x434e4c52; // "RLNC" + private const string Kind = "dueling-q"; + + public static PgSnakeNet Parse(Stream checkpoint) + { + using var r = new BinaryReader(checkpoint, Encoding.UTF8, leaveOpen: true); + if (r.ReadUInt32() != Magic) + throw new InvalidDataException("Not an RLNC checkpoint."); + string kind = r.ReadString(); + if (kind != Kind) + throw new InvalidDataException($"Expected checkpoint kind '{Kind}', got '{kind}'."); + + int version = r.ReadInt32(); + int inputSize = r.ReadInt32(); + int hiddenCount = r.ReadInt32(); + var hidden = new List(hiddenCount); + for (int i = 0; i < hiddenCount; i++) hidden.Add(r.ReadInt32()); + int actions = r.ReadInt32(); + bool noisy = version >= 2 && r.ReadByte() != 0; + + List ReadFloats() + { + int n = r.ReadInt32(); + var a = new List(n); + for (int i = 0; i < n; i++) a.Add(r.ReadSingle()); // float32 → its exact f64, matching the browser + return a; + } + + var trunkWFlat = new List(); + var trunkBFlat = new List(); + for (int l = 0; l < hiddenCount; l++) + { + trunkWFlat.AddRange(ReadFloats()); + trunkBFlat.AddRange(ReadFloats()); + } + + List valueW, valueB, advW, advB; + if (!noisy) + { + valueW = ReadFloats(); valueB = ReadFloats(); + advW = ReadFloats(); advB = ReadFloats(); + } + else + { + // Noisy heads store Mean/Sigma pairs; inference runs with noise off, so keep the Mean tensors only. + valueW = ReadFloats(); ReadFloats(); valueB = ReadFloats(); ReadFloats(); + advW = ReadFloats(); ReadFloats(); advB = ReadFloats(); ReadFloats(); + } + + return new PgSnakeNet(inputSize, actions, hidden, trunkWFlat, trunkBFlat, valueW, valueB, advW, advB); + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg index eb847d7..fe49b5a 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg @@ -83,6 +83,11 @@ class PgSnakeNet { } } +// A search-frontier node: a cloned env, the root move that first led here, and its leaf score. A record (typed list +// element) — NOT parallel Lists — so the transpiled TS types the beam nominally; parallel `List` frontiers make +// the emitted `let x = []` evolve to a circular `any` (childFirst ← beamFirst ← nextFirst ← childFirst → TS7022). +record PgSnakeBeamNode(env: PgSnakeEnv, firstMove: i32, score: f64) + // ── Snake environment (port of SnakeEnv.cs) ───────────────────────────────────────────────────────────────── class PgSnakeEnv { const ActionCount: i32 = 4 @@ -370,4 +375,167 @@ class PgSnakeEnv { } return best } + + // ── Look-ahead search (M34) ─────────────────────────────────────────────────────────────────────────────── + + // Deep copy of the live state for branching. safeMask is forced OFF: sims use reversal-only legality and let the + // survival SCORE — not a hard 1-ply shield — judge trapping, so the search can see, rank, and step through + // near-trap lines a shield would have pruned outright. body/occupied are copied element-wise (occupied already + // has `cells` entries from the ctor; body starts empty). + fn clone(): PgSnakeEnv { + let c = PgSnakeEnv(this.size, this.stepPenalty, false) + for x in this.body { c.body.add(x) } + for i in 0..this.cells { c.occupied[i] = this.occupied[i] } + c.food = this.food + c.foodEaten = this.foodEaten + c.heading = this.heading + c.elapsedSteps = this.elapsedSteps + c.stepsSinceFood = this.stepsSinceFood + c.done = this.done + return c + } + + // Deterministic food placement for simulated lines. The real env's caller owns the RNG (see `needsFood`); the + // search must stay RNG-free so C# and TS branch byte-identically. The exact fictional cell barely matters — + // survival scoring drives the search and every tick re-plans — so "first free cell" is a fine, stable choice. + fn simSpawnFood() { + for cell in 0..this.cells { + if !this.occupied[cell] { this.food = cell; return } + } + } + + // Max reachable open space over the 4 candidate next-head cells (the pre-Polyglot SnakeEnv.FreeSpaceAhead): the + // leaf survivability signal — a line that leaves no room for the body is a guaranteed future self-trap. + fn freeSpaceAhead(): i32 { + let head = this.headCell() + let hr = head / this.size + let hc = head % this.size + let tail = this.tailCell() + var best = 0 + for a in 0..ActionCount { + let v = this.reachableFreeSpace(hr + drOf(a), hc + dcOf(a), tail) + if v > best { best = v } + } + return best + } + + // First legal (non-reversal) move — the fallback when the horizon somehow yields no scored line. + fn firstLegalAction(): i32 { + let mask = this.currentActionMask() + for a in 0..ActionCount { if mask[a] { return a } } + return 0 + } + + // Score a simulated leaf just reached (this = the child after step). A board-full win dominates everything; a + // death is dominated by everything but ranked to prefer a LATER death (buys time for the tail to clear); a + // survived node scores food eaten − a hard trap penalty when the reachable region can no longer hold the body + // + free space + a food-proximity tiebreak. This is a PURE SURVIVAL score — the net does NOT enter here: a net + // forward at every node is ~9× the per-move cost and (measured, M34) buys no strength, forcing a shallower and + // thus WEAKER search. The trained net instead breaks ties between equally-safe ROOT moves in chooseActionSearch + // (one forward per move). Weights are passed in (not bundled in a record) so this file declares no top-level types. + fn leafScoreSearch(rootFood: i32, depth: i32, foodWeight: f64, trapPenalty: f64, spaceWeight: f64, foodDistWeight: f64, spaceRatioWeight: f64): f64 { + if this.lastTerminated && this.body.count == this.cells { return 1000000000.0 } + if this.lastTerminated { return -1000000.0 + f64(depth) * 1000.0 } + + let foodGained = this.foodEaten - rootFood + let free = this.freeSpaceAhead() + var score = f64(foodGained) * foodWeight + if free < this.body.count { score = score - trapPenalty } + score = score + f64(free) * spaceWeight + + // Connectivity term: fraction of the currently-free cells still reachable going forward. Unlike the absolute + // `free < len` trap penalty, this catches FRAGMENTATION the body still fits into — the snake cutting itself off + // from most of the board — which is the failure mode a fixed-window view misses. The biggest single lever found + // (M34): a heavy weight (~100k) lifts food@12 ~71 → ~81. weight 0 disables it. + if spaceRatioWeight != 0.0 { + let freeCells = this.freeCount() + if freeCells > 0 { score = score + spaceRatioWeight * f64(free) / f64(freeCells) } + } + + let head = this.headCell() + let hr = head / this.size + let hc = head % this.size + let fr = this.food / this.size + let fc = this.food % this.size + let foodDist = f64(iabs(fr - hr) + iabs(fc - hc)) + score = score - foodDist * foodDistWeight + return score + } + + // Net-tiebroken receding-horizon beam search — the M34 planner. Simulates every legal line to `maxDepth` plies on + // cloned envs (pure-survival leaf scoring), tracking the best line score achievable from each ROOT move; then + // plays the root move with the best line, breaking ties by the trained net's Q for that move from the current + // state. So the flood-fill survival search delivers the strength (depth is the lever) and the net decides "which + // of the equally-safe ways to go" — one forward per move, not per node. Re-planned every tick. The returned move + // is always a non-reversal, so a live env with safeMask on OR off accepts it (drive it with safeMask OFF — this + // planner supersedes the reactive 1-ply shield). + fn chooseActionSearch(net: PgSnakeNet, maxDepth: i32, beamWidth: i32, foodWeight: f64, trapPenalty: f64, netWeight: f64, spaceWeight: f64, foodDistWeight: f64, spaceRatioWeight: f64): i32 { + let rootFood = this.foodEaten + // Best line score reachable from each root move (−inf = that move was never expanded, i.e. illegal at the root). + var bestByRoot: List = List() + for a in 0..ActionCount { bestByRoot.add(-1000000000000.0) } + + let root = this.clone() + var beam: List = List() + beam.add(PgSnakeBeamNode(root, -1, 0.0)) + + for depth in 0..maxDepth { + if beam.count == 0 { break } + var next: List = List() + for node in beam { + let mask = node.env.currentActionMask() // reversal-only (the clone's safeMask is off) + for a in 0..ActionCount { + if !mask[a] { continue } + let child = node.env.clone() + child.step(a) + if child.needsFood { child.simSpawnFood() } + let childFirst = if node.firstMove < 0 { a } else { node.firstMove } + let score = child.leafScoreSearch(rootFood, depth + 1, foodWeight, trapPenalty, spaceWeight, foodDistWeight, spaceRatioWeight) + if score > bestByRoot[childFirst] { bestByRoot[childFirst] = score } + if !child.done && depth + 1 < maxDepth { next.add(PgSnakeBeamNode(child, childFirst, score)) } + } + } + beam = pruneBeam(next, beamWidth) + } + + // Pick the root move with the best line; the net's Q (one forward, from the current state) breaks near-ties so + // the trained model chooses among equally-safe directions. netWeight scales that nudge (0 ⇒ pure survival). + let rootQ = net.forward(this.buildObservation()) + var bestFinal = -1000000000000000.0 + var bestFirst = root.firstLegalAction() + for a in 0..ActionCount { + if bestByRoot[a] <= -1000000000000.0 { continue } + var v = bestByRoot[a] + if netWeight != 0.0 { v = v + rootQ[a] * netWeight } + if v > bestFinal { + bestFinal = v + bestFirst = a + } + } + return bestFirst + } + + // Keep the `k` highest-scoring nodes (all of them if ≤ k). Manual top-k scan — a List sort with a comparator + // doesn't transpile cleanly (same reason FruitCake's selectTopK scans instead of sorting). + static fn pruneBeam(nodes: List, k: i32): List { + if nodes.count <= k { return nodes } + var kept: List = List() + var used: List = List() + for i in 0..nodes.count { used.add(false) } + for _iter in 0..k { + var bestIdx = -1 + var bestVal = 0.0 + for i in 0..nodes.count { + if used[i] { continue } + if bestIdx < 0 || nodes[i].score > bestVal { + bestVal = nodes[i].score + bestIdx = i + } + } + if bestIdx < 0 { break } + used[bestIdx] = true + kept.add(nodes[bestIdx]) + } + return kept + } } diff --git a/src/RLDemo.Web/ClientApp/src/app/snake/snake-director.ts b/src/RLDemo.Web/ClientApp/src/app/snake/snake-director.ts index dfd21d4..9b54d9e 100644 --- a/src/RLDemo.Web/ClientApp/src/app/snake/snake-director.ts +++ b/src/RLDemo.Web/ClientApp/src/app/snake/snake-director.ts @@ -1,14 +1,28 @@ import { PgSnakeEnv, PgSnakeNet } from './snake_solver'; import { loadSnakeNet } from './snake-net'; -// Client-side "watch the AI" director — the whole Snake AI runs in the browser (M33). It drives the single-source -// PgSnakeEnv (dynamics + 177-dim observation + the anti-self-trap flood-fill action mask) and the masked-greedy -// chooseAction over the trained net loaded from the shipped checkpoint — no server, no WebSocket. Discrete-tick -// (one AI move per tick), so the component drives it on a plain interval, like human play. +// Client-side "watch the AI" director — the whole Snake AI runs in the browser (M33 + M34 search). It drives the +// single-source PgSnakeEnv (dynamics + 177-dim observation + flood-fill survivability) and the net-guided +// multi-ply look-ahead `chooseActionSearch` over the trained net loaded from the shipped checkpoint — no server, +// no WebSocket. The search is the lever that lifts play from the reactive ~50-food plateau to ~75+ (M34): it +// simulates every legal line and keeps the snake out of boxes it can't escape, with the net scoring the leaves. +// Discrete-tick (one AI move per tick), so the component drives it on a plain interval, like human play. const SIZE = 12; // the shipped net was trained on a 12×12 board -const SAFE_MASK = true; // the deployed net was trained WITH the flood-fill shield — inference must match -const STEP_PENALTY = -0.01; // training-only; irrelevant to greedy inference +const SAFE_MASK = false; // the planner's survival scoring supersedes the reactive 1-ply shield (it plans deeper) +const STEP_PENALTY = -0.01; // training-only; irrelevant to greedy/search inference + +// Net-tiebroken look-ahead tuning (M34). The flood-fill survival search does the heavy lifting; the net breaks ties +// between equally-safe moves (one forward per move). Depth 12 / beam 16 is the measured sweet spot — deeper/wider +// scored WORSE (beam pruning misranks deep lines, per PR #11's sweep) and slower. Mirrors SnakeSearchConfig in C#. +const SEARCH_DEPTH = 12; +const SEARCH_BEAM = 16; +const W_FOOD = 10_000; +const W_TRAP = 50_000; +const W_NET = 50; // small: the net only breaks ties between equally-safe root moves (a big weight slightly hurts — measured) +const W_SPACE = 50; +const W_DIST = 1; +const W_RATIO = 100_000; // anti-fragmentation: fraction of free cells still reachable. Biggest lever — ~71 → ~81 food@12 (M34) const DEAD_HOLD_TICKS = 8; // show the dead board briefly before auto-restarting export interface SnakeAiFrame { @@ -53,7 +67,7 @@ export class SnakeDirector { } if (this.net === null) return this.frame(); - const action = this.core.chooseAction(this.net); + const action = this.core.chooseActionSearch(this.net, SEARCH_DEPTH, SEARCH_BEAM, W_FOOD, W_TRAP, W_NET, W_SPACE, W_DIST, W_RATIO); if (action < 0) { this.deadHold = DEAD_HOLD_TICKS; return this.frame(); } // no legal move (shouldn't happen) this.core.step(action); if (this.core.needsFood) this.core.spawnFood(this.randFree()); diff --git a/src/RLDemo.Web/ClientApp/src/app/snake/snake.html b/src/RLDemo.Web/ClientApp/src/app/snake/snake.html index e69a31e..79f4dcd 100644 --- a/src/RLDemo.Web/ClientApp/src/app/snake/snake.html +++ b/src/RLDemo.Web/ClientApp/src/app/snake/snake.html @@ -1,8 +1,9 @@

Snake

- A masked Double + Dueling DQN that learned Snake from scratch — watch it play or - play it yourself, both running entirely in your browser (no server). - Honest about its skill: it seeks food and dodges walls, but a from-scratch net eventually traps itself. + A masked Double + Dueling DQN guided by a multi-ply look-ahead search — watch it play + or play it yourself, both running entirely in your browser (no server). + The learned net evaluates moves; a flood-fill survival search plans several steps ahead to avoid boxing itself in, + so it fills much more of the board than the net alone.

diff --git a/src/RLDemo.Web/ClientApp/src/app/snake/snake_solver.ts b/src/RLDemo.Web/ClientApp/src/app/snake/snake_solver.ts index 866549b..870eec4 100644 --- a/src/RLDemo.Web/ClientApp/src/app/snake/snake_solver.ts +++ b/src/RLDemo.Web/ClientApp/src/app/snake/snake_solver.ts @@ -1,4 +1,17 @@ export type Option = { tag: "Some"; value: T } | { tag: "None" }; +export class PgSnakeBeamNode { + env: PgSnakeEnv; + firstMove: number; + score: number; + constructor(env: PgSnakeEnv, firstMove: number, score: number) { + this.env = env; + this.firstMove = firstMove; + this.score = score; + } + equals(other: PgSnakeBeamNode): boolean { + return this.env === other.env && this.firstMove === other.firstMove && this.score === other.score; + } +} export class PgSnakeNet { inputSize: number; actions: number; @@ -408,4 +421,164 @@ export class PgSnakeEnv { } return best; } + clone(): PgSnakeEnv { + const c = new PgSnakeEnv(this.size, this.stepPenalty, false); + for (const x of this.body) { + c.body.push(x); + } + for (let i = 0; i < this.cells; i++) { + c.occupied[i] = this.occupied[i]; + } + c.food = this.food; + c.foodEaten = this.foodEaten; + c.heading = this.heading; + c.elapsedSteps = this.elapsedSteps; + c.stepsSinceFood = this.stepsSinceFood; + c.done = this.done; + return c; + } + simSpawnFood(): void { + for (let cell = 0; cell < this.cells; cell++) { + if (!this.occupied[cell]) { + this.food = cell; + return; + } + } + } + freeSpaceAhead(): number { + const head = this.headCell(); + const hr = (head / this.size | 0); + const hc = (head % this.size | 0); + const tail = this.tailCell(); + let best = 0; + for (let a = 0; a < PgSnakeEnv.ActionCount; a++) { + const v = this.reachableFreeSpace((hr + PgSnakeEnv.drOf(a) | 0), (hc + PgSnakeEnv.dcOf(a) | 0), tail); + if (v > best) { + best = v; + } + } + return best; + } + firstLegalAction(): number { + const mask = this.currentActionMask(); + for (let a = 0; a < PgSnakeEnv.ActionCount; a++) { + if (mask[a]) { + return a; + } + } + return 0; + } + leafScoreSearch(rootFood: number, depth: number, foodWeight: number, trapPenalty: number, spaceWeight: number, foodDistWeight: number, spaceRatioWeight: number): number { + if (this.lastTerminated && this.body.length === this.cells) { + return 1000000000.0; + } + if (this.lastTerminated) { + return -1000000.0 + depth * 1000.0; + } + const foodGained = (this.foodEaten - rootFood | 0); + const free = this.freeSpaceAhead(); + let score = foodGained * foodWeight; + if (free < this.body.length) { + score = score - trapPenalty; + } + score = score + free * spaceWeight; + if (spaceRatioWeight !== 0.0) { + const freeCells = this.freeCount(); + if (freeCells > 0) { + score = score + spaceRatioWeight * free / freeCells; + } + } + const head = this.headCell(); + const hr = (head / this.size | 0); + const hc = (head % this.size | 0); + const fr = (this.food / this.size | 0); + const fc = (this.food % this.size | 0); + const foodDist = (PgSnakeEnv.iabs((fr - hr | 0)) + PgSnakeEnv.iabs((fc - hc | 0)) | 0); + score = score - foodDist * foodDistWeight; + return score; + } + chooseActionSearch(net: PgSnakeNet, maxDepth: number, beamWidth: number, foodWeight: number, trapPenalty: number, netWeight: number, spaceWeight: number, foodDistWeight: number, spaceRatioWeight: number): number { + const rootFood = this.foodEaten; + let bestByRoot = []; + for (let a = 0; a < PgSnakeEnv.ActionCount; a++) { + bestByRoot.push(-1000000000000.0); + } + const root = this.clone(); + let beam = []; + beam.push(new PgSnakeBeamNode(root, (-1 | 0), 0.0)); + for (let depth = 0; depth < maxDepth; depth++) { + if (beam.length === 0) { + break; + } + let next = []; + for (const node of beam) { + const mask = node.env.currentActionMask(); + for (let a = 0; a < PgSnakeEnv.ActionCount; a++) { + if (!mask[a]) { + continue; + } + const child = node.env.clone(); + child.step(a); + if (child.needsFood) { + child.simSpawnFood(); + } + const childFirst = (node.firstMove < 0 ? a : node.firstMove); + const score = child.leafScoreSearch(rootFood, (depth + 1 | 0), foodWeight, trapPenalty, spaceWeight, foodDistWeight, spaceRatioWeight); + if (score > bestByRoot[childFirst]) { + bestByRoot[childFirst] = score; + } + if (!child.done && (depth + 1 | 0) < maxDepth) { + next.push(new PgSnakeBeamNode(child, childFirst, score)); + } + } + } + beam = PgSnakeEnv.pruneBeam(next, beamWidth); + } + const rootQ = net.forward(this.buildObservation()); + let bestFinal = -1000000000000000.0; + let bestFirst = root.firstLegalAction(); + for (let a = 0; a < PgSnakeEnv.ActionCount; a++) { + if (bestByRoot[a] <= -1000000000000.0) { + continue; + } + let v = bestByRoot[a]; + if (netWeight !== 0.0) { + v = v + rootQ[a] * netWeight; + } + if (v > bestFinal) { + bestFinal = v; + bestFirst = a; + } + } + return bestFirst; + } + static pruneBeam(nodes: PgSnakeBeamNode[], k: number): PgSnakeBeamNode[] { + if (nodes.length <= k) { + return nodes; + } + let kept = []; + let used = []; + for (let i = 0; i < nodes.length; i++) { + used.push(false); + } + for (let _iter = 0; _iter < k; _iter++) { + let bestIdx = (-1 | 0); + let bestVal = 0.0; + for (let i = 0; i < nodes.length; i++) { + if (used[i]) { + continue; + } + if (bestIdx < 0 || nodes[i].score > bestVal) { + bestVal = nodes[i].score; + bestIdx = i; + } + } + if (bestIdx < 0) { + break; + } + used[bestIdx] = true; + kept.push(nodes[bestIdx]); + } + return kept; + } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs index 5ba6c5b..7be234e 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs @@ -1,7 +1,9 @@ +using System.Diagnostics; using System.Globalization; using Microsoft.Extensions.DependencyInjection; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; +using MintPlayer.AI.ReinforcementLearning.Environments.Snake; using MintPlayer.AI.ReinforcementLearning.Hosting; /// @@ -29,6 +31,26 @@ public static void Run(string[] args) float stepPenalty = -0.01f; // --step-penalty : per-step reward; ~0 removes the efficiency pressure that encourages safe starvation bool safeMask = false; // --safe-mask : forbid moves that flood-fill into a region too small for the body (anti-self-trap shield) bool evalOnly = false; + + // --search : skip training and evaluate the net-guided look-ahead planner (M34) instead of greedy Q. The + // net is only a leaf tiebreak, so the config defaults reproduce PR #11's shipped depth-20/beam-32 sweep. + bool search = false; + string netPath = Path.Combine("src", "RLDemo.Web", "wwwroot", "models", "snake-net.ckpt"); + var cfg = new SnakeSearchConfig(); + for (int i = 0; i < args.Length; i++) + { + if (args[i] == "--search") search = true; + else if (args[i] == "--net" && i + 1 < args.Length) netPath = args[++i]; + else if (args[i] == "--depth" && i + 1 < args.Length) cfg = cfg with { MaxDepth = int.Parse(args[++i]) }; + else if (args[i] == "--beam" && i + 1 < args.Length) cfg = cfg with { BeamWidth = int.Parse(args[++i]) }; + else if (args[i] == "--w-food" && i + 1 < args.Length) cfg = cfg with { FoodWeight = double.Parse(args[++i], CultureInfo.InvariantCulture) }; + else if (args[i] == "--w-trap" && i + 1 < args.Length) cfg = cfg with { TrapPenalty = double.Parse(args[++i], CultureInfo.InvariantCulture) }; + else if (args[i] == "--w-net" && i + 1 < args.Length) cfg = cfg with { NetWeight = double.Parse(args[++i], CultureInfo.InvariantCulture) }; + else if (args[i] == "--w-space" && i + 1 < args.Length) cfg = cfg with { SpaceWeight = double.Parse(args[++i], CultureInfo.InvariantCulture) }; + else if (args[i] == "--w-dist" && i + 1 < args.Length) cfg = cfg with { FoodDistWeight = double.Parse(args[++i], CultureInfo.InvariantCulture) }; + else if (args[i] == "--w-ratio" && i + 1 < args.Length) cfg = cfg with { SpaceRatioWeight = double.Parse(args[++i], CultureInfo.InvariantCulture) }; + } + for (int i = 0; i < args.Length; i++) { if (args[i] == "--hours" && i + 1 < args.Length) hours = double.Parse(args[++i], CultureInfo.InvariantCulture); @@ -48,6 +70,12 @@ public static void Run(string[] args) else if (args[i] == "--eval-only") evalOnly = true; } + if (search) + { + RunSearchEval(netPath, evalGrid, evalEpisodes, seed, cfg); + return; + } + // DI all the way: the model store, clock and CampaignRunner are resolved from the AIHost container. using var host = AIHost.CreateBuilder(dataDir).Build(); var store = host.Services.GetRequiredService(); @@ -63,4 +91,53 @@ public static void Run(string[] args) OnEval = CampaignCli.ConsoleAndCsv(csvPath), }); } + + /// + /// Evaluates the net-guided look-ahead planner (M34) over games on a + /// × board, reporting the food distribution and per-move latency + /// (the latter is what gates the in-browser client-side director). The env runs with safeMask: false — + /// the planner's survival scoring supersedes the reactive 1-ply shield. + /// + private static void RunSearchEval(string netPath, int grid, int episodes, ulong seed, SnakeSearchConfig cfg) + { + if (!File.Exists(netPath)) + { + Console.Error.WriteLine($"Checkpoint not found: {Path.GetFullPath(netPath)} (pass --net )."); + return; + } + + var env = new SnakeEnv(grid, safeMask: false); + using (var stream = File.OpenRead(netPath)) + env.LoadSearchNet(stream); + + Console.WriteLine($"Search eval: {episodes} episodes on {grid}×{grid}, net {Path.GetFileName(netPath)}"); + Console.WriteLine($" config: depth={cfg.MaxDepth} beam={cfg.BeamWidth} food={cfg.FoodWeight} trap={cfg.TrapPenalty} net={cfg.NetWeight} space={cfg.SpaceWeight} dist={cfg.FoodDistWeight} ratio={cfg.SpaceRatioWeight}"); + + int totalFood = 0, maxFood = 0, minFood = int.MaxValue; + long totalMoves = 0; + var sw = Stopwatch.StartNew(); + for (int ep = 0; ep < episodes; ep++) + { + env.Reset(seed + (ulong)ep); + bool done = false; + while (!done) + { + int action = env.ChooseActionSearch(cfg); + var step = env.Step(action); + done = step.Terminated || step.Truncated; + totalMoves++; + } + int food = env.FoodEaten; + totalFood += food; + maxFood = Math.Max(maxFood, food); + minFood = Math.Min(minFood, food); + Console.WriteLine($" ep {ep + 1,3}: food {food}"); + } + sw.Stop(); + + double meanFood = (double)totalFood / episodes; + double msPerMove = totalMoves == 0 ? 0 : sw.Elapsed.TotalMilliseconds / totalMoves; + Console.WriteLine($"food@{grid}: mean {meanFood:F1} (min {minFood}, max {maxFood}, {episodes} eps)"); + Console.WriteLine($"planner latency: {msPerMove:F1} ms/move ({totalMoves} moves, {sw.Elapsed.TotalSeconds:F1}s)"); + } }