From dc89ee5f7455ee3d94030d2d7763744003a8cec8 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 5 Jul 2026 20:46:55 +0200 Subject: [PATCH 1/9] docs(m33): PRD+plan for client-side Snake & MountainCar AI; bump Polyglot 0.1.4->0.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 3-agent investigation + Polyglot 0.3.0 verification. 0.3.0 fixes all of #11 (transcendentals added, module imports now link, i32() TS wrap fixed) and #9 stays fixed — so MountainCar's transcendental fork dissolves (cos/tanh writable in a .pg; sub-ULP C#<->TS drift harmless post-cutover). FruitCake TS codegen verified content-identical on 0.3.0 (22 tests green). Filed remaining gaps as MintPlayer.Polyglot#11 (all addressed in 0.3.0). Plan: Snake first (clean full-Polyglot port), MountainCar second (now also uniform Polyglot). Co-Authored-By: Claude Opus 4.8 --- .../CLIENTSIDE_AI_SNAKE_MOUNTAINCAR_PRD.md | 188 ++++++++++++++++++ docs/prd/PLAN.md | 23 +++ ....ReinforcementLearning.Environments.csproj | 2 +- 3 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 docs/prd/CLIENTSIDE_AI_SNAKE_MOUNTAINCAR_PRD.md diff --git a/docs/prd/CLIENTSIDE_AI_SNAKE_MOUNTAINCAR_PRD.md b/docs/prd/CLIENTSIDE_AI_SNAKE_MOUNTAINCAR_PRD.md new file mode 100644 index 0000000..ce490e5 --- /dev/null +++ b/docs/prd/CLIENTSIDE_AI_SNAKE_MOUNTAINCAR_PRD.md @@ -0,0 +1,188 @@ +# Client-side AI for Snake & MountainCar (PRD & Plan) — M33 + +> **Goal (user's steer):** do for the remaining WebSocket-AI games what M32 did for FruitCake — move the env +> **dynamics** + **observation** + **net forward** (+ any serving "wrapper" like masking) into a MintPlayer.Polyglot +> `.pg` where feasible, ship the trained **checkpoint** as a browser asset, run the AI **client-side**, and delete +> the server serving path. Per-viewer server inference/streaming → **zero**. + +- **Status:** Draft v1.0 · 2026-07-05 (3-agent investigation + Polyglot 0.2.0 probes complete; not started) +- **Author:** Pieterjan (with Claude Code) +- **Precedent:** `FRUITCAKE_CLIENT_SIDE_AI_PRD.md` (M32, shipped) — the pattern this replicates. + +--- + +## 1. Scope — exactly two games + +The only games whose AI "watch" mode holds an open WebSocket are **Snake** (`/api/snake/live`) and **MountainCar** +(`/api/mountaincar/live`), both driven by the shared `EpisodeStreamer`. The other three are **request/response REST** +batch solvers with no per-viewer socket, so they need nothing here: +- **2048** — `POST /api/2048/solve` (expectimax, one response). +- **RushHour** — `POST /api/rushhour/solve` (policy-guided A*, one response). +- **Cube** — `POST /api/cube/solve*` (beam search, one response). + +So M33 = **Snake + MountainCar**, as **two independent PRs** (they share only `EpisodeStreamer` and the checkpoint +parser). **Snake first** (clean, high-value, lowest new-code); MountainCar second (has a real complication — §5). + +--- + +## 2. What we confirmed (3-agent investigation + Polyglot 0.2.0 probes, 2026-07-05) + +### Polyglot 0.3.0 (verified 2026-07-05 — supersedes the 0.2.0 notes below) +**0.3.0 fixes all of #11** (verified by probe): **transcendentals added** (`cos`/`sin`/`tan`/`exp`/`log`/`pow`/`tanh` +— emit `System.Math.Cos`/`Math.cos` etc.); **module imports now LINK** (TS emits `import {…} from "./nn"`, C# no +longer inlines → a shared `nn.pg` is viable, no CS0101); **`i32()` TS wrap fixed** (`Math.trunc(x)`, no `|0`). #9 +stays fixed. **Consequence: MountainCar's transcendental fork DISSOLVES** — `cos`/`tanh` can now be written directly +in a `.pg`; the C#↔TS sub-ULP drift is harmless (no server twin post-cutover, argmax-robust), so MountainCar gets the +same uniform full-Polyglot treatment as Snake (no poly-approx). §5's Option A is now the clean default. The +Environments `.csproj` is bumped 0.1.4 → **0.3.0** (FruitCake TS codegen verified content-identical, 22 tests green). + +### Polyglot 0.2.0 (historical) +- **All three M32 codegen bugs are FIXED** (issue MintPlayer.Polyglot#9, closed): `i32()/f64()` casts now lower to + `(int)x` / `Math.trunc(x)|0` / `(double)n`; `var x: T? = null` emits a typed decl; 2+ nested-generic params parse. + ⇒ **new `.pg` files can use casts, nullable locals, and nested generics freely** (no M32 workarounds needed). +- **Integer `/` and `%` codegen correctly** (`a/b` → `(a/b|0)` in TS; `a%b` → `(a%b|0)`) — Snake's grid index math + (`head/Size`, `head%Size`, non-negative) is clean. +- **std.math has NO transcendentals.** Only `abs`, `floor`, `sqrt` (+ `min`/`max`) exist; **`cos`, `sin`, `exp`, + `pow`, `tanh` are absent** — consistent with "only `+ − × ÷ √` are byte-identical across .NET and JS." This is the + crux for MountainCar (§5). +- **User `.pg`→`.pg` imports DO work** (`import { X } from "./nn"` and `from "nn"` both check + build) — but via + **inlining**: the importer's output gets a *full copy* of the imported symbols (no cross-module reference emitted). + In the MSBuild "glob every `**/*.pg` → compile all into one assembly" setup, a standalone library `.pg` would then + be defined **twice** (once from its own transpile, once inlined into the importer) → CS0101 duplicate type. A shared + `nn.pg` is therefore feasible only if the library file is **excluded from the C# transpile glob** (and not committed + as a standalone TS). ⇒ **For two games, duplicate the net-forward per game** — Snake copies `PgDuelingNet` verbatim + (~70 lines), MountainCar gets a small `PgMlpNet`; the tiny duplication is simpler than the build-config work a + shared module needs. (Revisit a shared `nn.pg` only if a third+ net-bearing game appears.) + +### Checkpoint parser generalization +The `.ckpt` envelope (`CheckpointFormat`) is shared: `RLNC` magic + 7-bit-length kind string + int32 version, then +length-prefixed int/float/bool blocks. Two payload shapes: +- **`dueling-q`** (Snake) — identical to FruitCake. **`fruitcake-net.ts` parses it unchanged.** +- **`mlp`** (MountainCar) — `Ints Sizes` + `byte HiddenActivation` + per-layer `(W, b)` floats. +⇒ Lift the shared header/primitive readers out of `fruitcake-net.ts` into a **shared `ckpt.ts`** (`parseDuelingQNet` ++ a new `parseMlp`); each game's loader hands the parsed tensors to *its* generated net class. + +### EpisodeStreamer +`src/RLDemo.Web/Services/EpisodeStreamer.cs` is used **only** by Snake + MountainCar → **deletable** once both +migrate (delete in whichever PR lands second). + +--- + +## 3. Snake — full Polyglot client-side port (the clean one) + +**Facts** (`Environments/Snake/SnakeEnv.cs`; net `snake.dqn.ckpt`): +- **Observation = 177** (a 9×9×2 egocentric patch = 162, + 15 scalars). *(Note: the "8-ray obs" memory is stale — + it's the 177-wide patch.)* Pure integer/`+−×÷` math → **byte-identity is free**. +- **Net = plain `DuelingQNet` 177 → [256,256] → 4**, checkpoint kind `dueling-q` v1 → **reuse `PgDuelingNet.forward` + and the existing parser verbatim.** +- **No search** — one greedy, **masked** Q-step per tick. Much simpler than FruitCake. +- **The "wrapper" that must move to the browser = the action mask** (`CurrentActionMask` + `ReachableFreeSpace`): + (a) forbid the 180° reversal onto the neck; (b) the **anti-self-trap shield** — a **BFS flood-fill** keeping only + moves whose reachable free space ≥ snake length (never all-false). The net was trained *with* this shield + (`safeMask:true`), so serving must reproduce it exactly. +- **Collections to re-express** (the main porting effort): `SnakeEnv` uses `LinkedList` (body), `HashSet` + (occupancy), `Queue` (BFS) — Polyglot's `std.collections` is `List` only. Re-express as: body = + `List` used as a deque (add-front / remove-last); occupancy = `List` of size `Cells`; BFS = `List` + + head cursor, `seen` = `List`. All flat, all integer. +- **Food RNG** → `Math.random` client-side (nondeterministic, fine for a watch view; no need to port Xoshiro). +- **Rendering** is a CSS-grid of `
`s (not canvas). Director `toFrame()` → `{body, food, foodEaten, done, + length}`, so the existing `render()` is unchanged. **Discrete tick** — reuse a `setInterval`/timer (like human + mode), not a RAF physics loop. + +### Snake plan (SN0–SN6) +- **SN0 — `snake_solver.pg` scaffold + env dynamics.** Port `SnakeEnv` step (move/grow/collision/food, tail-follow + ordering) into `PgSnakeEnv` over flat `List` collections; bump the Environments `.csproj` Polyglot ref to **0.2.0**. + Gate: builds; a C# facade delegates; parity test vs `SnakeEnv` step outcomes. +- **SN1 — observation in the `.pg`.** `buildObservation() -> f64[177]` (9×9×2 patch + 15 scalars + the flood-fill + free-space features). Gate: reproduces `SnakeEnv.Observation()` exactly (integer math → exact, not tolerance). +- **SN2 — action mask in the `.pg`.** `currentActionMask() -> [bool;4]` (reversal + flood-fill shield). Gate: matches + `SnakeEnv.CurrentActionMask()` on played boards. +- **SN3 — net forward.** Copy `PgDuelingNet` into `snake_solver.pg` (verbatim from FruitCake). Gate: argmax-exact vs + SDK `DuelingQNet.Forward` (reuse the M32 parity-test pattern). +- **SN4 — checkpoint delivery + shared parser.** Copy `models/snake.dqn.ckpt` → `ClientApp/public/snake-net.ckpt` + (LFS); refactor `fruitcake-net.ts` → shared `ckpt.ts` (`parseDuelingQNet` unchanged). Gate: browser parses it → + 177/[256,256]/4. +- **SN5 — client director + rewire watch mode.** `snake-director.ts`: masked greedy step over `PgSnakeEnv` + + `PgDuelingNet`, `toFrame()` for the existing renderer; rewrite `snake.ts` watch mode to drive it (drop + `snake-api.ts`). Reconcile human-play `snake-logic.ts` onto the single-source `PgSnakeEnv` (eliminate the twin). + Gate: host + Playwright — Snake AI plays in-browser, 0 console errors, 0 `/api/snake` calls. +- **SN6 — retire server path.** Delete `SnakeController`, `SnakeModelService` (+ `Program.cs` regs), `snake-api.ts`, + stale `models/snake.dqn.ckpt`; update `SnakeApiTests`. Gate: build + tests green; Playwright watch still works. + +--- + +## 4. Reusable M32 machinery (both games) + +Build-time C# transpile via `MintPlayer.Polyglot.MSBuild` (bump **0.1.4 → 0.2.0**); committed generated `.ts` +(regenerate on Windows with `polyglot build … --target typescript`); the internal-core + public-facade pattern + +`InternalsVisibleTo` for tests; **`.ckpt` committed directly to `ClientApp/public/` via LFS** (M32 did *not* use an +MSBuild copy — the asset is committed); the async-load client **director**; and the **Polyglot-C# == SDK-C# +equivalence tests** (+ C#↔TS byte-identity free for integer/`+−×÷√` code). + +--- + +## 5. MountainCar — the transcendental fork (needs a decision) + +MountainCar is tiny (env ≈ 15 lines; net = `Mlp [2,64,64,3]`), but it has **two transcendentals on the client path**, +and **neither can be written in a `.pg`** (std.math has no `cos`/`tanh`): +1. **`cos(3·position)`** in the env dynamics (every step). +2. **`tanh`** — the PPO actor's hidden activation (`Mlp`, `Activation.Tanh`). Serving = `argmax` over 3 logits. + +(The observation itself — `(pos+0.3)/0.9`, `vel/MaxSpeed` — is pure `+−×÷`, so the *net input* is byte-safe; the +checkpoint is kind `mlp`, needing a new `parseMlp` + a `PgMlpNet` forward.) + +> **UPDATE (Polyglot 0.3.0):** `cos` and `tanh` are now in std.math, so both CAN be written in a `.pg`. They are not +> byte-identical across C#/TS (~1 ULP), but that no longer matters (no server twin after cutover; argmax-robust). So +> **Option A is now the clean default — no polynomial approximations needed.** The fork below is retained for record. + +So MountainCar **cannot** get FruitCake's "byte-identical single source" for free. Two viable approaches: + +- **Option A — uniform Polyglot, with byte-identical polynomial approximations of `cos` and `tanh`** (pure `+−×÷`). + *Pro:* true single-source + C#↔TS byte-identity, matching the uniform goal. *Con:* the approximations change the + numbers vs the trained float32 `Math.Cos`/`Math.Tanh`, so it needs a **validation gate** (argmax unchanged across a + position/velocity sweep vs the real net) — feasible if the approximations are accurate (~1e-6), but real work and a + small risk of needing a re-eval/retrain. +- **Option B — pragmatic client-side port, no Polyglot for the transcendental parts (RECOMMENDED).** Reuse the + existing hand-written `mountaincar-logic.ts` dynamics (it already has `Math.cos`), hand-write the ~30-line + `Mlp`+`tanh` forward in TS (JS `Math.tanh` — a ULP off .NET, argmax-robust), add `parseMlp`, ship the ckpt, delete + the socket. *Pro:* achieves the actual goal (zero server cost, client-side AI) with minimal work and **no net + perturbation**. *Con:* keeps a ~15-line C#/TS env twin (but MountainCar dynamics are stable and never change, so + drift risk is negligible), and doesn't "single-source via Polyglot." + +**Recommendation:** **Option B.** For a 15-line env whose twin never changes, Polyglot single-sourcing buys almost +nothing, while the transcendentals actively fight it. The cost win (removing the socket) is identical either way. +Reserve Option A only if *uniform Polyglot single-source* is a hard requirement — in which case gate it behind an +approximation-accuracy spike (**MC0**) before committing. + +### MountainCar plan (MC0–MC5) — assuming Option B (adjust if A is chosen) +- **MC0 (only if Option A) — approximation spike.** Implement `cos`/`tanh` in pure `+−×÷`; verify argmax matches the + real net across a state sweep. Pass → proceed Polyglot (mirror Snake's SN plan). Fail → fall back to Option B. +- **MC1 — `parseMlp` in the shared `ckpt.ts`** (Sizes + activation byte + per-layer W/b). +- **MC2 — TS `Mlp` forward** (`mountaincar-net.ts`: linear + tanh per hidden layer, linear head, argmax). Gate: + matches SDK `Mlp.Forward`/`PolicyAgent.Act` argmax on a state sweep. +- **MC3 — checkpoint delivery.** Copy `models/mountaincar.ppo.ckpt` → `ClientApp/public/mountaincar-net.ckpt` (LFS). +- **MC4 — client director + rewire.** `mountaincar-director.ts`: obs → net → argmax → step (reuse `mountaincar-logic.ts` + dynamics) → render; rewrite `mountaincar.ts` watch mode (drop `mountaincar-api.ts`). Gate: Playwright — plays, + 0 console errors, 0 `/api/mountaincar` calls. +- **MC5 — retire server path + delete `EpisodeStreamer`.** Delete `MountainCarController`, `MountainCarModelService` + (+ regs), `mountaincar-api.ts`, `EpisodeStreamer.cs` (now unused), stale `models/mountaincar.ppo.ckpt`; update + `MountainCarApiTests`. Gate: build + tests green; Playwright watch works. + +--- + +## 6. Risks + +| Risk | Game | Mitigation | +|---|---|---| +| Action-mask shield (flood-fill) must be reproduced exactly in the browser | Snake | Port `CurrentActionMask` + `ReachableFreeSpace` to the `.pg`; equivalence-test vs `SnakeEnv`. Integer math → exact. | +| Collection re-expression (LinkedList/HashSet/Queue → List) introduces a bug | Snake | Parity test the ported step + obs + mask against `SnakeEnv` on many played boards. | +| `cos`/`tanh` can't be in a `.pg` → no byte-identity | MountainCar | Option B (recommended) sidesteps it (hand-port, argmax-robust); Option A gates on an accuracy spike. | +| Net perturbation from approximations | MountainCar (Opt A only) | MC0 spike validates argmax parity before committing. | +| Deploy: LFS asset must reach the SPA build | both | Same as M32 (verified): `playground-docker.yml` `lfs:true` + Dockerfile `COPY ClientApp/` before `npm build`; Angular serves `public/**`. | + +--- + +## 7. Non-goals +- 2048 / RushHour / Cube (request/response, no socket). +- Retraining Snake or MountainCar (ship the existing checkpoints; both play well today). +- A shared `nn.pg` (user `.pg` imports unsupported — duplicate per game). diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 082350d..83b6ae2 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1103,6 +1103,29 @@ in parallel). **Supersedes PG4.** Also advances MintPlayer.Polyglot (FruitCake i > architecture):** the shipped browser net is the **G3** net; if its net+search play quality (G4 A/B vs the > ~50%-watermelon / ~2505 bar) is unsatisfying, retrain (M30/G4) and replace `ClientApp/public/fruitcake-net.ckpt`. > Still recommended: **split into separate PRs** (NetTransfer is independently mergeable; M30/M31/M32 are entangled). +> **UPDATE: merged to master via PR #23 (single PR) 2026-07-05.** A/B ship-as-is (2493 / 49% watermelon, on par). + +## M33 — Client-side AI for Snake & MountainCar *(planned — see `CLIENTSIDE_AI_SNAKE_MOUNTAINCAR_PRD.md`)* 🔜 + +Extend M32 to the **only two remaining WebSocket-AI games** (2048/RushHour/Cube are request/response REST → out of +scope). A 3-agent investigation + Polyglot **0.2.0** probes (2026-07-05) confirmed: 0.2.0 **fixed all three M32 +codegen bugs** (#9 closed — casts / null-typed-locals / nested-generics now work); std.math still has **no +transcendentals** (only `abs`/`floor`/`sqrt`/`min`/`max`); **user `.pg`→`.pg` imports work but *inline* the imported +symbols** (a standalone library `.pg` would double-define under the glob-all build) → **duplicate the net-forward per +game** (simpler than the glob-exclusion a shared `nn.pg` needs); the `.ckpt` parser generalizes to one shared `ckpt.ts` +(`dueling-q` + a new `mlp` branch); +`EpisodeStreamer` is used only by these two → deletable when both migrate. **Two independent PRs, Snake first.** +- **Snake — clean full-Polyglot port.** obs **177** (9×9×2 patch + 15 scalars; the "8-ray" memory is stale); net = + plain `DuelingQNet 177→[256,256]→4` → **reuses `PgDuelingNet` + the parser verbatim**; **no search** (one greedy + masked Q-step). Real work: port the **action mask** (reversal + anti-self-trap **flood-fill shield**, load-bearing — + net trained with it) and re-express `LinkedList`/`HashSet`/`Queue` as flat `List`s. Pure integer math → byte-identity + free. Plan SN0–SN6. +- **MountainCar — transcendental fork.** Two transcendentals on the client path (`cos(3·pos)` in the env, **`tanh`** + in the PPO `Mlp` net) — **neither writable in a `.pg`**. **Recommended: Option B** — pragmatic client-side hand-port + (reuse `mountaincar-logic.ts` dynamics, ~30-line TS `Mlp`+`tanh` forward, new `parseMlp`, ship ckpt, delete socket); + no net perturbation, keeps a trivial ~15-line env twin. **Option A** (uniform Polyglot with byte-identical `cos`/`tanh` + polynomial approximations) only if uniform single-source is a hard requirement, gated on an argmax-parity spike (MC0). + Plan MC0–MC5. **Payoff:** the last two per-viewer AI sockets → zero server inference. ## Testing strategy (cross-cutting, from research) diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index f4c13d4..e46ccef 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -9,7 +9,7 @@ build-only transpiler OUT of this published package's dependencies. The 0.1.3+ CLI is bundled for win-x64/linux-x64/linux-arm64 (covers Windows dev + Linux CI/deploy); macOS dev must set $(PolyglotTool). --> - + + + + + diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg new file mode 100644 index 0000000..1d0562e --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/polyglot/snake_solver.pg @@ -0,0 +1,364 @@ +// snake_solver.pg — single-source Snake env + inference (M33), transpiled to C# and TS. +// +// ONE source for the game dynamics + observation + action mask (incl. the anti-self-trap flood-fill "shield") +// + the dueling-Q forward pass, so the C# training/eval env AND the browser "watch AI" run identical logic. +// f64 for the net; the env is pure integer + `+ − × ÷` math, so C# and TS are byte-identical. +// +// Collections: Polyglot's std.collections List has add / removeAt / [i] get+set / count / clear (no +// removeFirst/removeLast/insert). So the snake body is a List with the HEAD AT THE END (add = grow head, +// removeAt(0) = drop tail); occupancy is a List of size cells (occupied.count == body.count always); the +// BFS uses a List + a cursor and a List seen-set. No HashSet/LinkedList/Queue. +// +// The generated C# is transpiled at build (MintPlayer.Polyglot.MSBuild); the TS is committed +// (snake_solver.ts), regenerated on Windows: polyglot build snake_solver.pg --target typescript --out + +import { List } from "std.collections" +import { Math } from "std.math" + +// ── Dueling-Q net inference (copied verbatim from fruitcake_solver.pg; Snake's net is the same architecture) ── +// f64 forward pass over flat weight arrays. Q(s,a) = A(s,a) + (V(s) − mean_a A(s,a)); ReLU trunk; linear heads. +// Weights (float32 in the .ckpt) are loaded into these f64 arrays by the per-platform parser. +class PgDuelingNet { + var inputSize: i32 + var actions: i32 + var hidden: List + var trunkWFlat: List + var trunkBFlat: List + var valueW: List + var valueB: List + var advW: List + var advB: List + + init(inputSize: i32, actions: i32, hidden: List, trunkWFlat: List, trunkBFlat: List, valueW: List, valueB: List, advW: List, advB: List) { + this.inputSize = inputSize + this.actions = actions + this.hidden = hidden + this.trunkWFlat = trunkWFlat + this.trunkBFlat = trunkBFlat + this.valueW = valueW + this.valueB = valueB + this.advW = advW + this.advB = advB + } + + fn forward(obs: List): List { + var x = obs + var prev = this.inputSize + var wOff = 0 + var bOff = 0 + for l in 0..this.hidden.count { + let h = this.hidden[l] + x = relu(linear(x, this.trunkWFlat, wOff, this.trunkBFlat, bOff, prev, h)) + wOff += prev * h + bOff += h + prev = h + } + let value = linear(x, this.valueW, 0, this.valueB, 0, prev, 1) + let adv = linear(x, this.advW, 0, this.advB, 0, prev, this.actions) + + var sumA = 0.0 + for k in 0..this.actions { sumA += adv[k] } + let meanA = sumA / this.actions + let v = value[0] + + var q: List = List() + for k in 0..this.actions { q.add(adv[k] + (v - meanA)) } + return q + } + + static fn linear(x: List, w: List, wOff: i32, b: List, bOff: i32, inDim: i32, outDim: i32): List { + var out: List = List() + for o in 0..outDim { + var s = b[bOff + o] + for i in 0..inDim { s += x[i] * w[wOff + i * outDim + o] } + out.add(s) + } + return out + } + + static fn relu(x: List): List { + var out: List = List() + for v in x { out.add(Math.max(0.0, v)) } + return out + } +} + +// ── Snake environment (port of SnakeEnv.cs) ───────────────────────────────────────────────────────────────── +class PgSnakeEnv { + const ActionCount: i32 = 4 + const PatchRadius: i32 = 4 + const PatchSide: i32 = 9 // 2*PatchRadius+1 + const PatchPlane: i32 = 81 // PatchSide*PatchSide + const PatchSize: i32 = 162 // PatchPlane*2 (obstacle + food channels) + const ObservationSize: i32 = 177 // PatchSize + 15 scalars + const FoodReward: f64 = 1.0 + const DeathReward: f64 = -1.0 + + var size: i32 + var cells: i32 + var stepPenalty: f64 + var safeMask: bool + var body: List // head at END: body[count-1]=head, body[0]=tail + var occupied: List // size cells; occupied.count invariant == body.count + var food: i32 + var foodEaten: i32 + var heading: i32 + var elapsedSteps: i32 + var stepsSinceFood: i32 + var done: bool + // Step outputs (read by the C# training facade; the browser director only needs `done` + state). + var lastReward: f64 + var lastTerminated: bool + var lastTruncated: bool + + init(size: i32, stepPenalty: f64, safeMask: bool) { + this.size = size + this.cells = size * size + this.stepPenalty = stepPenalty + this.safeMask = safeMask + this.body = List() + this.occupied = List() + for i in 0..this.cells { this.occupied.add(false) } + this.food = 0 + this.foodEaten = 0 + this.heading = 3 + this.elapsedSteps = 0 + this.stepsSinceFood = 0 + this.done = true + this.lastReward = 0.0 + this.lastTerminated = false + this.lastTruncated = false + } + + // Action deltas: Up(0)=(-1,0) Down(1)=(1,0) Left(2)=(0,-1) Right(3)=(0,1). + static fn drOf(a: i32): i32 { if a == 0 { return -1 } if a == 1 { return 1 } return 0 } + static fn dcOf(a: i32): i32 { if a == 2 { return -1 } if a == 3 { return 1 } return 0 } + static fn iabs(x: i32): i32 { if x < 0 { return 0 - x } return x } + + fn headCell(): i32 => this.body[this.body.count - 1] + fn neckCell(): i32 => this.body[this.body.count - 2] + fn tailCell(): i32 => this.body[0] + let length: i32 => this.body.count + fn freeCount(): i32 => this.cells - this.body.count + + // Reset to a length-3 horizontal snake, centred, head pointing Right. Food is spawned separately by the caller + // (spawnFood) so the RNG stays out of the single source: C# rolls Xoshiro, the browser rolls Math.random. + fn reset() { + this.body.clear() + for i in 0..this.cells { this.occupied[i] = false } + let row = this.size / 2 + let headCol = this.size / 2 + // head-at-end: add tail → head, i.e. c = headCol-2, headCol-1, headCol. + for c in (headCol - 2)..(headCol + 1) { + let cell = row * this.size + c + this.body.add(cell) + this.occupied[cell] = true + } + this.heading = 3 + this.foodEaten = 0 + this.elapsedSteps = 0 + this.stepsSinceFood = 0 + this.done = false + } + + // Place food at the pick'th free cell (pick in [0, freeCount)). Caller supplies the random index. + fn spawnFood(pick: i32) { + var p = pick + for cell in 0..this.cells { + if this.occupied[cell] { continue } + if p == 0 { this.food = cell; return } + p = p - 1 + } + } + + // Advance one step with the given action (assumed legal — call currentActionMask() first). foodPick is the + // random free-cell index used only if this move eats and the board isn't full. + fn step(action: i32, foodPick: i32) { + let head = this.headCell() + let headRow = head / this.size + let headCol = head % this.size + let newRow = headRow + drOf(action) + let newCol = headCol + dcOf(action) + + if newRow < 0 || newRow >= this.size || newCol < 0 || newCol >= this.size { + this.die() + return + } + let newHead = newRow * this.size + newCol + let eating = newHead == this.food + let tail = this.tailCell() + if this.occupied[newHead] && !(newHead == tail && !eating) { + this.die() + return + } + + this.body.add(newHead) // grow head (head-at-end) + this.heading = action + var terminated = false + if eating { + this.occupied[newHead] = true + this.foodEaten = this.foodEaten + 1 + this.stepsSinceFood = 0 + this.lastReward = FoodReward + if this.body.count == this.cells { + terminated = true // board full — a win + } else { + this.spawnFood(foodPick) + } + } else { + // Remove the vacating tail BEFORE marking the new head occupied (tail-follow ordering; see SnakeEnv.cs). + this.occupied[tail] = false + this.body.removeAt(0) + this.occupied[newHead] = true + this.stepsSinceFood = this.stepsSinceFood + 1 + this.lastReward = this.stepPenalty + } + + this.elapsedSteps = this.elapsedSteps + 1 + let starveLimit = 2 * this.cells + let maxSteps = 100 * this.cells + let truncated = !terminated && (this.stepsSinceFood >= starveLimit || this.elapsedSteps >= maxSteps) + this.done = terminated || truncated + this.lastTerminated = terminated + this.lastTruncated = truncated + } + + fn die() { + this.done = true + this.lastReward = DeathReward + this.lastTerminated = true + this.lastTruncated = false + } + + // A cell is an obstacle if off-board or a non-vacating body segment (the tail vacates on a non-eating move). + fn obstacle(r: i32, c: i32, tail: i32): bool { + if r < 0 || r >= this.size || c < 0 || c >= this.size { return true } + let cell = r * this.size + c + return this.occupied[cell] && cell != tail + } + + fn freeCell(r: i32, c: i32, tail: i32): bool { + if r < 0 || r >= this.size || c < 0 || c >= this.size { return false } + let cell = r * this.size + c + return !this.occupied[cell] || cell == tail + } + + // BFS flood-fill: count of free cells reachable from (r,c). Bounded at `cells` iterations (each cell processed + // once). List queue + cursor instead of a Queue; List seen instead of a HashSet. + fn reachableFreeSpace(r: i32, c: i32, tail: i32): i32 { + if !this.freeCell(r, c, tail) { return 0 } + var seen: List = List() + for i in 0..this.cells { seen.add(false) } + var queue: List = List() + let start = r * this.size + c + seen[start] = true + queue.add(start) + var cursor = 0 + var count = 0 + for _iter in 0..this.cells { + if cursor >= queue.count { break } + let cell = queue[cursor] + cursor = cursor + 1 + count = count + 1 + let cr = cell / this.size + let cc = cell % this.size + for a in 0..ActionCount { + let nr = cr + drOf(a) + let nc = cc + dcOf(a) + if !this.freeCell(nr, nc, tail) { continue } + let n = nr * this.size + nc + if seen[n] { continue } + seen[n] = true + queue.add(n) + } + } + return count + } + + // The 177-dim observation (port of SnakeEnv.Observation): 9×9×2 egocentric patch (channel-major) + 15 scalars. + fn buildObservation(): List { + let head = this.headCell() + let hr = head / this.size + let hc = head % this.size + let tail = this.tailCell() + let fr = this.food / this.size + let fc = this.food % this.size + + var obs: List = List() + for i in 0..ObservationSize { obs.add(0.0) } + + var i = 0 + for dr in (0 - PatchRadius)..(PatchRadius + 1) { + for dc in (0 - PatchRadius)..(PatchRadius + 1) { + let r = hr + dr + let c = hc + dc + if (dr != 0 || dc != 0) && this.obstacle(r, c, tail) { obs[i] = 1.0 } + if r == fr && c == fc { obs[PatchPlane + i] = 1.0 } + i = i + 1 + } + } + + var s = PatchSize + obs[s] = f64(fc - hc) / f64(this.size); s = s + 1 + obs[s] = f64(fr - hr) / f64(this.size); s = s + 1 + obs[s] = f64(iabs(fr - hr) + iabs(fc - hc)) / (2.0 * f64(this.size)); s = s + 1 + obs[s] = if this.heading == 0 { 1.0 } else { 0.0 }; s = s + 1 + obs[s] = if this.heading == 1 { 1.0 } else { 0.0 }; s = s + 1 + obs[s] = if this.heading == 2 { 1.0 } else { 0.0 }; s = s + 1 + obs[s] = if this.heading == 3 { 1.0 } else { 0.0 }; s = s + 1 + obs[s] = f64(this.body.count) / f64(this.cells); s = s + 1 + for a in 0..ActionCount { + obs[s] = f64(this.reachableFreeSpace(hr + drOf(a), hc + dcOf(a), tail)) / f64(this.cells); s = s + 1 + } + let tr = tail / this.size + let tc = tail % this.size + obs[s] = f64(tc - hc) / f64(this.size); s = s + 1 + obs[s] = f64(tr - hr) / f64(this.size); s = s + 1 + obs[s] = f64(iabs(tr - hr) + iabs(tc - hc)) / (2.0 * f64(this.size)); s = s + 1 + return obs + } + + // Legal-move mask: forbid the 180° reversal onto the neck; with safeMask, also forbid moves that seal the snake + // into a region smaller than its length (flood-fill shield), never returning an all-false mask. + fn currentActionMask(): List { + var mask: List = List() + for a in 0..ActionCount { mask.add(true) } + if this.body.count < 2 { return mask } + let head = this.headCell() + let neck = this.neckCell() + let headRow = head / this.size + let headCol = head % this.size + for a in 0..ActionCount { + let r = headRow + drOf(a) + let c = headCol + dcOf(a) + if r >= 0 && r < this.size && c >= 0 && c < this.size && r * this.size + c == neck { mask[a] = false } + } + if !this.safeMask { return mask } + + let tail = this.tailCell() + let len = this.body.count + var safe: List = List() + for a in 0..ActionCount { safe.add(mask[a]) } + var any = false + for a in 0..ActionCount { + if !mask[a] { continue } + if this.reachableFreeSpace(headRow + drOf(a), headCol + dcOf(a), tail) >= len { any = true } + else { safe[a] = false } + } + if any { return safe } + return mask + } + + // The full single-source decision: masked greedy argmax over the net's Q-values (mirrors GreedyQAgent.Act). + fn chooseAction(net: PgDuelingNet): i32 { + let q = net.forward(this.buildObservation()) + let mask = this.currentActionMask() + var best = -1 + for a in 0..ActionCount { + if !mask[a] { continue } + if best < 0 { best = a } + else if q[a] > q[best] { best = a } + } + return best + } +} From aeabdfa8887b438e7b9cd5020758512ee4ed93de Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 5 Jul 2026 21:58:06 +0200 Subject: [PATCH 3/9] =?UTF-8?q?feat(m33):=20unblock=20Snake=20.pg=20on=20P?= =?UTF-8?q?olyglot=200.3.1=20(#14=20fixed)=20=E2=80=94=20builds=20alongsid?= =?UTF-8?q?e=20FruitCake?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bump 0.3.0->0.3.1 (shared __polyglot_prelude.cs + partial PolyglotProgram fix #14), drop the PolyglotFile Remove workaround. Fixes to snake_solver.pg: rename PgDuelingNet->PgSnakeNet (distinct name; a deliberate copy of FruitCake's net — one assembly, so a shared name would clash; a shared nn.pg is a future refactor), and rename a fill-loop var to avoid a C# i-scope clash (CS0136). Environments builds with both .pg; 22 FruitCake/Polyglot tests green. Co-Authored-By: Claude Opus 4.8 --- ...layer.AI.ReinforcementLearning.Environments.csproj | 11 +---------- .../Snake/polyglot/snake_solver.pg | 8 ++++---- 2 files changed, 5 insertions(+), 14 deletions(-) diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index 70784b7..c81fcff 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -9,16 +9,7 @@ build-only transpiler OUT of this published package's dependencies. The 0.1.3+ CLI is bundled for win-x64/linux-x64/linux-arm64 (covers Windows dev + Linux CI/deploy); macOS dev must set $(PolyglotTool). --> - - - - - - +