diff --git a/CLAUDE.md b/CLAUDE.md index 59ce5bb..614ae86 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,40 +5,36 @@ Project-level instructions loaded every session. Read `docs/ARCHITECTURE.md` for ## Running / verifying the web app (RLDemo.Web) — READ THIS FIRST -**To build, serve, run, or visually verify the frontend, do exactly one thing:** - -```bash -dotnet run --project src/RLDemo.Web # Development profile; http://localhost:5210 -``` +**The .NET host is ALREADY RUNNING. The user runs and owns it. Do NOT start, stop, kill, restart, or +`taskkill` it — ever. Assume it is up at http://localhost:5210 and serving the Angular frontend.** The ASP.NET Core host **builds and serves the Angular frontend itself** — in Development it spawns and -proxies the Angular dev server (`UseAngularCliServer` via `UseSpaImproved` in `Program.cs`). There is -**nothing else to do for the frontend.** +proxies the Angular dev server (`UseAngularCliServer` via `UseSpaImproved` in `Program.cs`). The Angular app +is NOT a separate process you manage; it lives inside the running .NET host. -- **Never** run `ng serve` / `npm start` / `ng build` / `ng test` yourself. The host already runs one; a - second instance just fights for ports and can wedge the dev-server file watcher. +- **Do NOT run `dotnet run --project src/RLDemo.Web` yourself** — it is already running. A second instance + fights for the port. (This command is how the *user* starts it; it is not your job.) +- **Never** run `ng serve` / `npm start` / `ng build` / `ng test`, and never `taskkill`/kill the `dotnet` / + `RLDemo.Web.exe` / `node` (ng serve) processes. - **To see a code change:** just save the file under `ClientApp/src` — the running host live-reloads the - browser. No manual build, usually no manual reload. + browser. No manual build, no restart, usually no manual reload. - **To verify what's actually served** (suspected staleness): `curl -sk http://localhost:5210/main.js | grep ` - — do not reach for `ng build` to "check". -- If output looks stale, suspect a wedged dev-server watcher: **restart the ASP.NET host**, never `ng build`. + — do not reach for `ng build`, and do not restart the host, to "check". +- **If output looks stale** (wedged dev-server watcher) **or new npm dependencies were added** (the host must + re-read them): **ASK THE USER to restart their host.** Do not restart it yourself. ## Build failures are usually NOT the frontend If `dotnet run --project src/RLDemo.Web` fails, read the error before touching anything Angular-related. -**Known: stale Polyglot codegen (`CS0260` "missing partial modifier on PolyglotProgram" / `CS0101` -duplicate `Option`/`Some`/`None` prelude).** This is the documented multi-`.pg` incremental-rebuild -transpiler bug (see `docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`) — the CLI doesn't clean its -`--out` dir, so a stale duplicate lingers in `obj/`. It is unrelated to the frontend and to your changes. -Fix by clearing the stale generated files, then run again: - -```bash -rm -f src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs -dotnet run --project src/RLDemo.Web -``` - -Clean/CI builds are unaffected. Never loop build retries against the stale output. +**Fixed (2026-07-13, Polyglot 0.6.0): the multi-`.pg` incremental-rebuild codegen bug** (`CS0260` "missing +partial modifier on PolyglotProgram" / `CS0101` duplicate `Option`/`Some`/`None` prelude). It was an MSBuild +`.targets` bug — a single-`.pg` edit made MSBuild's partial-incremental build hand the transpiler a subset, +which then emitted a standalone/duplicate prelude. `MintPlayer.Polyglot.MSBuild` **0.6.0** (PR #26, stamp +`Outputs` + `RemoveDir`) re-transpiles the full `.pg` set on any edit, so this no longer occurs. If you somehow +hit it on an **older** package, the manual recovery was +`rm -f src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs` then rebuild. History + +root cause: `docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`. Never loop build retries against stale output. ## Stopping a `dotnet run` host diff --git a/docs/prd/CHESS_SELFPLAY_PRD.md b/docs/prd/CHESS_SELFPLAY_PRD.md new file mode 100644 index 0000000..00bdc1e --- /dev/null +++ b/docs/prd/CHESS_SELFPLAY_PRD.md @@ -0,0 +1,114 @@ +# Self-play training (chess) — PRD + +**Status:** Planned · 2026-07-12 · branch TBD (off `master`) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M39 · **Depends on:** the Core NN + checkpoint layer (§2/§5 of [../ARCHITECTURE.md](../ARCHITECTURE.md)), the two-headed `PolicyValueNet` (M37), the training-campaign harness (§8), and action masking (§3). Additive — no change to existing training behaviour. + +## 1. Problem + +The SDK trains agents in three paradigms today — model-free RL (DQN/PPO), planning over a learned value net (DAVI), and **imitation from an oracle** (the cube's Kociemba teacher, Rush Hour's BFS teacher). All three need an external signal: a reward function, a forward model, or an *exact oracle*. For a game like **chess** there is no cheap exact oracle, and a reactive policy plateaus — this repo's own history is unambiguous that **search is the lever** (Snake/FruitCake/Cube all show a capped reactive net amplified massively by inference search). + +The missing paradigm is **self-play**: two players (the same improving network) play each other, and the games themselves become the training signal — the network bootstraps from nothing to strong play with no human data and no oracle. That is how a from-scratch SDK trains chess. We want to add self-play as a **first-class, reusable capability** — chess is the headline, but the machinery (a two-player game abstraction + MCTS + a self-play campaign) should be shared so the *next* two-player game plugs in by writing only its rules. + +**Constraint (load-bearing):** reuse the SDK; write as little new code as possible. A three-agent investigation (2026-07-12) established that **~70% of the outer loop already exists** and the genuinely-new code is small and well-isolated (see §5). + +## 2. Goal & success criteria + +- **Self-play works, verifiably (the priority).** Starting from a **randomly-initialized** net, self-play training produces a net whose strength **rises monotonically** against a fixed baseline — measured first on **Connect-4** (converges in minutes on CPU; a negamax oracle gives an exact yardstick), then on **chess** (win-rate vs a random-legal baseline climbs; later, Elo vs a frozen earlier checkpoint). +- **Reuse-first.** The self-play stack reuses `PolicyValueNet` (unchanged), the soft-CE+value training step (already used by the imitation campaigns), `ITrainingCampaign`/`CampaignRunner`, the model store + checkpoint format, the M38 Lab plumbing (`AdamState`/`TrainWindow`/`PolicyGrowth`), action masking, the RNG streams, `--viz`, and the paired-seed A/B harness (→ the Elo promotion gate). New code is confined to: one game seam, MCTS, the self-play data loop, and each game's rules. +- **One new deep seam, not a shallow one.** `IZeroSumGame` in `Core/Planning` is the single new abstraction; it must pass the M38 bar (minimal, honest, consumed by *both* MCTS and the self-play trainer; Connect-4/chess/checkers/Go plug in unchanged). It is a **sibling** to `IDeterministicModel`, not an extension of it (see §3). +- **Correctness gated.** Chess legal-move generation is verified by **perft** (leaf-node counts matched to published values, depths 5–6, from startpos + Kiwipete + standard positions) *before any training*. Nothing downstream is trusted until perft passes. +- **Determinism & resume.** Self-play RNG (Dirichlet noise, move sampling) uses its own `SeedSequence` stream; a campaign resumes bitwise-consistently through the model store, like every other campaign (a `CampaignContractTests` roundtrip proves it). + +**Honest non-goals for v1 (strength).** Not superhuman, not engine-strength chess. The net is **MLP-only** (no conv/attention in the backend), so a flattened board loses spatial structure — this proves the pipeline and gives *steadily-improving, fully-legal* play, but caps positional strength. And a small chess MLP does **not** clear the GPU routing threshold that helps the cube, so self-play is CPU-bound. The realistic target is *"plays fully-legal, steadily-improving chess"* — Connect-4 is where the self-improvement curve is unmistakable; chess is the headline consumer of the same rails. Conv-backend support and true engine strength are explicitly out of scope (separate workstreams). + +**Other non-goals (v1).** No DQN/PPO for self-play (wrong fit — a ~4600-wide masked Q-head over terminal-only sparse reward under self-play non-stationarity); no reuse of the DQN `ReplayBuffer` (it's `(s,a,r,s′,term,mask)`-shaped; self-play wants an `(obs,π,z)` window — a plain list, like the imitation campaigns); no bitboards unless a bench says movegen is the bottleneck; no web showcase page in phase 1. + +## 3. Key decision — AlphaZero-style, on one new game seam, Connect-4 first + +**Design it twice** (full analysis in the investigation): + +- **Design A — plain self-play over `IEnvironment`.** Model the game side-to-move, reward = terminal outcome, opponent = the current/frozen net (a small `IOpponentPolicy` seam), train with PPO/REINFORCE. *Rejected as the endpoint:* no lookahead → tactically-blind play, the exact reactive plateau the repo documents. Useful only as a plumbing pipe-test. +- **Design B — AlphaZero-style (chosen).** MCTS-guided self-play: each move runs PUCT simulations whose leaves are scored by the two-headed net; the move is sampled from the **root visit-count distribution** π; training targets are `(π, z)` where `z` is the game outcome from that position's mover. Loss = `CE(π, policyLogits) + MSE(z, tanh(value))`. + +**Why B, given "minimal new code":** the chess rules engine is the *same irreducible cost* in both designs. B's only marginal cost over A is `Mcts.cs` + visit-count targets instead of an opponent seam + PPO glue — and B is the design that makes "two players play each other and the model improves" actually *true* for a tactically deep game. It also showcases the SDK's signature composition (game model + net + search), which is the repo's whole thesis. + +**Why a new seam, not `IEnvironment` or `IDeterministicModel`:** `IEnvironment` is single-agent (single scalar reward, episode loop) — it can host Design A but not MCTS. `IDeterministicModel` is *close* — its non-mutating `Apply` is exactly right — but its `IsGoal` (single bool) can't express **win/loss/draw relative to the side to move**, its `ActionCount` (valid in every state) lies for chess's per-state legal moves, and it carries no observation/policy-size. Bolting those on would leak (a shallow, dishonest interface). So add a minimal sibling: + +```csharp +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// Terminal result RELATIVE TO THE SIDE TO MOVE in the queried state. +public enum GameResult { Ongoing, Win, Loss, Draw } + +/// +/// A perfect-information, two-player, zero-sum, deterministic game — the shared shape MCTS and a self-play +/// trainer consume. Distinct from IEnvironment (no reward/episode loop) and IDeterministicModel (side-to-move +/// + win/loss/draw + per-state legal moves, not a single goal). Apply returns a FRESH successor and leaves +/// 'state' untouched (search reuses a state across moves); after Apply the side to move has flipped. +/// +public interface IZeroSumGame +{ + int PolicySize { get; } // fixed action-index space = the net's policy-head width + int ObservationSize { get; } + TState Root(ulong? seed = null); + IReadOnlyList LegalMoves(TState state); + TState Apply(TState state, int move); // non-mutating; flips side to move + GameResult Result(TState state); // terminal result for the mover, or Ongoing + void WriteObservation(TState state, Span destination); // side-to-move-relative encoding +} +``` + +**Why Connect-4 first:** it de-risks the *novel* machinery (MCTS soundness, value-sign backup, self-play target generation, non-collapse) on a game with trivial, fast, easily-correct rules and a cheap **negamax** oracle for an exact test — *separately* from chess's large, silent-bug-prone rules surface. Chess then lands as the seam's **second consumer**, riding the same MCTS/campaign/net/checkpoint rails unchanged. This is the same "prove the mechanism cheaply, then scale" discipline the rest of the repo follows, and each piece stays independently verifiable. + +## 4. Design — the pieces + +| Piece | Where | New/Reused | +|---|---|---| +| `IZeroSumGame` + `GameResult` | `Core/Planning/IZeroSumGame.cs` | **new** (deep seam) | +| `Mcts` — PUCT search: select → expand-with-priors → sign-flipping value backup; Dirichlet root noise; temperature; returns the root visit-count π | `Core/Planning/Mcts.cs` | **new** (~300–450 LOC; the core novel algorithm — no tree search exists in the repo) | +| `PolicyValueTraining.TrainStep(net, adam, obs[], policyTargets[], valueTargets[], batch)` — soft-CE + value regression, grad-clip, Adam | `Core/Training/` (generalized from `CubePolicyTraining.TrainStep`) | **new-ish** (~60 LOC; cube-imitation can also call it) | +| `SelfPlayCampaign : ITrainingCampaign, INetworkTelemetrySource` — `TrainChunk` plays K self-play games (MCTS both sides via net leaf) → `(obs,π,z)` rolling window → train → `PolicyGrowth.Maybe`; `Evaluate` = arena win-rate vs frozen best (→ Elo); `Checkpoint` = save-best on win-rate + `AdamState.Save` | `tools/…Lab/` | **new** (mirrors `CubeImitationCampaign`; reuses `AdamState`/`TrainWindow`/`PolicyGrowth`/telemetry) | +| `PolicyValueNet` — the AlphaZero net, **unchanged**; value wrapped in `tanh` at the call site for WDL in [-1,1] | `Core/Nn/PolicyValueNet.cs` | **reused** | +| `Connect4Game : IZeroSumGame<…>` + a negamax oracle (test) | `Environments/Connect4/` | **new** (small) | +| `ChessBoard` + legal movegen + draw/mate detection; move encoding (8×8×73 = 4672); `ChessGame : IZeroSumGame` / `ChessEnv`; plane observation; `ChessPolicyNet` wrapper; perft tests | `Environments/Chess/` | **new** (the large, test-heavy chunk) | +| Elo eval (`…Ab`), `--game connect4|chess` Lab dispatch, checkpoints via the model store + Git LFS | reuse patterns | **reused** | + +**MCTS composition** mirrors `ValueGuidedSearch` over `IDeterministicModel`: + +```csharp +public static class Mcts +{ + public sealed record Config(int Simulations = 400, float Cpuct = 1.25f, + float DirichletAlpha = 0.3f, float RootNoiseFrac = 0.25f); + + /// Leaf evaluator: priors over PolicySize + value in [-1,1] for the mover (PolicyValueNet under NoGrad). + public delegate (float[] Priors, float Value) Evaluate(TState state); + + /// Search from 'state'; return the root visit-count distribution over PolicySize (the training target π). + public static float[] Search(IZeroSumGame game, TState state, + Evaluate evaluate, Config config, Xoshiro256StarStar rng); +} +``` + +## 5. Reuse-vs-new ledger (rough) + +**Reused verbatim (0 new lines):** `PolicyValueNet` + Adam + tape autograd + `LogSoftmax`/`Mul`/`Sum`/`Tanh`; `ITrainingCampaign`/`CampaignRunner`/`AIHost`; `IModelStore`/`FileModelStore` + `CheckpointFormat` + `PolicyValueNet.Save/Load` (generic over a `kind` string, v2 trunk-widths); `IEnvironment`/`Space`/`IActionMaskProvider`/`IStatefulEnvironment`; `Net2Net` growth; `--viz`; the A/B harness pattern; `SeedSequence` streams. + +**New:** the game seam + MCTS + the `PolicyValueTraining` step + the self-play campaign (~900–1100 LOC total, most of it MCTS + the campaign) — plus each game's rules: Connect-4 (small), **chess (~1,500–2,200 LOC, rules + perft dominate)**. Total ≈ 3,000–4,500 LOC, of which the chess rules engine is the bulk and MCTS is the subtle part. + +## 6. Risks + +1. **Chess-rule correctness (highest).** Castling (through/into check), en passant (incl. discovered check), promotion, pins, and draw rules (50-move, threefold via position hashing, insufficient material, stalemate vs checkmate) are silent-bug territory. **Mitigation, non-optional:** perft node-count tests to depth 5–6 from standard positions matched exactly to published counts, *before* any training. Isolating this in M39.2 (chess) — after the rails are proven on Connect-4 in M39.1 — means a movegen bug can't masquerade as an MCTS/self-play bug. +2. **MCTS soundness / self-play collapse.** Wrong value-sign backup, missing Dirichlet root noise, or no temperature schedule → collapsing targets. Mitigation: unit-test MCTS on forced-win/forced-draw Connect-4 positions against negamax; gate training on rising win-rate vs a *frozen* baseline. +3. **Compute.** Self-play games are long; MCTS is hundreds of sims/move. CPU-bound (the small MLP won't clear the GPU threshold). Mitigation: Connect-4 for the convergence proof; keep chess sims modest; batched-leaf MCTS is a phase-3 lever. Set expectations: legal, improving play — not engine strength. +4. **Representation ceiling.** MLP over flattened planes has no spatial prior. Accepted for v1; conv backend is a separate workstream if positional strength becomes a goal. +5. **Robustness / exploitability — the "a novel move disorients the AI" failure (M39.3 lever).** Two nets trained only against each other co-adapt into a narrow shared distribution; off-distribution positions (a human's weird/suboptimal move) are undertrained, so the value/policy net misjudges them and — with too few sims — the agent can lose to a much weaker player. This is real even at superhuman level (the KataGo cyclic-group exploit let amateurs beat a superhuman Go bot). **Primary defence is already in the design:** at inference the agent runs **MCTS from the actual position** (it computes, it doesn't recall), so it generalizes to novel positions far better than a reactive net — this is a core reason we chose AlphaZero over reactive self-play. **Coverage levers (M39.3):** (a) **Dirichlet root noise + temperature sampling** — *already in the M39.1 MCTS + campaign* — make self-play explore off-policy lines so the training distribution doesn't collapse; (b) **opponent-pool / league play** — train against a pool of past checkpoints and occasionally a **random/weak mover**, the direct fix for "punish an unexpected blunder" and for co-adaptation; (c) **diverse/randomized opening positions**; (d) enough inference sims + **adversarial fine-tune** on any discovered exploit. Note: **NoisyNets is *not* the right tool here** — it's a DQN weight-noise exploration trick that perturbs the policy globally but does not ensure the agent *visits* diverse positions; AlphaZero exploration is Dirichlet-noise + temperature + opponent diversity, which target coverage directly. + +## 7. Verification + +- **M39.1 gate (Connect-4):** perft-free (tiny rules, unit-tested directly); MCTS unit tests vs negamax on forced positions; from random init, self-play win-rate vs negamax/random **climbs**; `CampaignContractTests` resume roundtrip (fresh → `TrainChunk` advances → `Checkpoint` → new instance `Resume`s and continues); `dotnet test --filter "Category!=Slow"` green. +- **M39.2 gate (chess):** **perft matches published counts** (startpos, Kiwipete, …) to depth 5–6 — the hard gate; move-encode/decode round-trips over all legal moves; env terminated-vs-truncated split correct; win-rate vs random-legal climbs; contract-test resume; ship `models/chess.az.ckpt` (LFS). +- Every step ends on a green build + its gate, revert-friendly, one milestone at a time. + +See [PLAN.md](PLAN.md) M39 for the phased step order. diff --git a/docs/prd/CHESS_WEB_POLYGLOT_PRD.md b/docs/prd/CHESS_WEB_POLYGLOT_PRD.md new file mode 100644 index 0000000..83fca90 --- /dev/null +++ b/docs/prd/CHESS_WEB_POLYGLOT_PRD.md @@ -0,0 +1,209 @@ +# Play the chess AI in the browser — single-source via MintPlayer.Polyglot — PRD + +**Status:** Planned · 2026-07-12 · branch TBD (off the M39 branch / `master`) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M40 · **Depends on:** M39 (the chess engine, the self-play-trained `PolicyValueNet`, the `.ckpt` format) and the MintPlayer.Polyglot single-source toolchain already used by FruitCake ([../ARCHITECTURE.md](../ARCHITECTURE.md) §10; `Environments/FruitCake/polyglot/`). + +## 1. Problem + +You can *watch* a game the chess AI played (a replay), but you can't *play against it in the browser* — there's no web chess page, and the AI is C# (net + MCTS), so a static page can't run it. The wrong fix is a server `ChessController` that runs MCTS per move (server inference, per-viewer CPU — the exact thing M32/M33 removed for FruitCake/Snake). The **right** fix is the pattern MintPlayer.Polyglot exists for and FruitCake already proves: **write the inference path once in a `.pg`, transpile to C# (training/serving) and TypeScript (browser), and run the whole AI client-side, with the browser downloading and parsing the `.ckpt`.** Zero server inference. + +## 2. Feasibility — verified + +The open question was whether chess's inference math and move-gen fit Polyglot's subset. Probed by transpiling (2026-07-12): + +- **`Math.exp`, `Math.tanh`, `Math.log`, `Math.sqrt` all transpile** (with `import { Math } from "std.math"`) — so MCTS's masked-softmax priors, the `tanh` value head, and the PUCT `sqrt` term are all expressible. (FruitCake avoided transcendentals because its physics must be *bit-exact* across C#/JS; only `+ - * / sqrt` are. **Chess inference does not need bit-exactness** — the browser AI just needs to play well, not match C# to the ULP — so non-bit-exact `exp`/`tanh` are fine here.) +- **Bitwise operators `& | << >>` transpile** — castling-rights flags and any bit tricks work (booleans are also an option). +- **Polyglot itself is extendable if a gap appears.** The language/toolchain lives at `C:\Repos\MintPlayer.Polyglot`; a missing feature (e.g. a `match`/`switch`, an enum, a Math function) can be added there and PR'd. The known limits to design around (from the FruitCake solver): **no nested-generic params** (`List>` is mishandled — use flat `List` + offsets, as `PgDuelingNet` does), and prefer `i32` constants + `if/else` over enums/`switch` unless we add them upstream. + +**Conclusion:** the chess engine + observation + net forward + MCTS can be single-sourced to `chess_solver.pg` and run client-side. This is a build, not a research risk. + +## 3. Goal & success criteria + +- **Play a full game against the self-taught AI in the browser** — an interactive board (you're White; click/drag to move, legal moves highlighted), the AI (Black) replies from a `.pg`-single-sourced net + MCTS running **in your browser**, over the shipped `.ckpt` it downloads. No server move computation. +- **Single source.** The chess **engine** (legal move-gen, apply, terminal, move encoding), the **observation**, the **net forward**, and the **MCTS** live once in `chess_solver.pg`, transpiled to C# (compiled into the Environments assembly, like FruitCake) and to committed TypeScript for the web client. A parity test pins the C# facade to the generated core; **perft still passes 25/25 against the generated engine** (the M39.2 gate carries over onto the single source). +- **Zero server inference / load-only web app.** The web app only serves the `.ckpt` (static file, `application/octet-stream` mapping like the FruitCake net) and the SPA. Per-viewer server cost is zero — consistent with M32/M33. +- **Honest scope.** It's the M39 network — a small MLP, briefly CPU-trained — so it plays *legal, still-learning* chess, beatable by a decent human. Browser MCTS is tuned (modest sims) for interactive latency (~1–2 s/move). +- **Training unchanged.** Training stays on the SDK (autograd/GEMM); the `.pg` is **inference + rules** only. The engine (generated C#) is shared by training's self-play; the net *forward* in the `.pg` is inference-only (training learns the weights via autograd, exactly as FruitCake). + +**Non-goals.** Engine strength; a server chess endpoint; bit-exact C#/JS transcendentals (not needed — this isn't a shared fairness sim); underpromotion-free or reduced rules (the engine is already full + perft-verified); training in the browser. + +## 4. Design — follow the FruitCake pattern + +| Piece | Where | Note | +|---|---|---| +| `chess_solver.pg` | `Environments/Chess/polyglot/` | **single source:** the engine (a `Pg`-prefixed board + `legalMoves`/`apply`/`result`/`isAttacked`), `writeObservation`, `PgPolicyValueNet.forward` (flat-array trunk + policy/value heads, `tanh` value), the 4672 `encode`/`decode`, and a chess `mcts` (PUCT, masked-softmax priors, `sqrt` exploration, **no Dirichlet** — inference only). Transpiled to C# (`obj/`, build-time) + committed `chess_solver.ts`. | +| C# facade | `Environments/Chess/ChessGame.cs` (rewired) | Wraps the generated `Pg` engine as `IZeroSumGame` (float/host view), so M39's `SelfPlayCampaign`/`ChessGame` keep working on the single source. Replaces the hand-written `ChessBoard.cs` engine internals; **perft tests re-point at the generated core**. | +| TS `.ckpt` parser | `ClientApp/src/app/chess/chess-net.ts` | Mirrors the C# `PolicyValueNet` reader (magic `RLNC`, kind `selfplay-pv`, v2 trunk-widths, params in `Parameters()` order) → builds the generated `PgPolicyValueNet`. The one non-Polyglot piece (binary I/O), exactly like `fruitcake-net.ts`. | +| Chess director | `ClientApp/src/app/chess/chess-director.ts` | Runs the transpiled `mcts` + net over the loaded `.ckpt` to pick the AI's reply — the analog of `fruit-cake-director.ts`. | +| Angular page | `ClientApp/src/app/chess/` + route/nav | Interactive board (unicode glyphs or simple SVG), human move input validated by the transpiled `legalMoves`, AI reply from the director, status (check/mate/draw), last-move highlight. | +| Shipped weights | `wwwroot/models/chess.az.ckpt` (LFS) + the `.ckpt` MIME mapping | Fetched by the browser; the checkpoint the director loads. | + +## 5. Phased plan (each ends on a green build + its gate) + +- **M40.1 — single-source the engine.** Port `ChessBoard`/`ChessRules` + `ChessMoveEncoding` into `chess_solver.pg`; wire the Polyglot MSBuild transpile; rewire `ChessGame`/facade onto the generated core; **re-run perft (25/25) + the encoding round-trip on the generated engine.** De-risks the biggest port (rules) first; training self-play still passes its contract test. +- **M40.2 — single-source the inference math + TS parser.** Add `PgPolicyValueNet.forward` (flat arrays, `tanh`), `writeObservation`, and a chess `mcts` to the `.pg`; a `chess-net.ts` `.ckpt` parser; a parity test (C# `PolicyValueNet.Forward` vs the generated `PgPolicyValueNet` on a fixed position, within f32 tolerance). Emit the committed `chess_solver.ts`. +- **M40.3 — the browser page.** `chess-director.ts` + the Angular chess component (board, input, AI reply, status) + route/nav; ship `wwwroot/models/chess.az.ckpt`; the `.ckpt` static-file mapping. **Gate:** play a full legal game vs the AI in the browser with no server move calls (verified via the network panel / Playwright); tune sims for latency. A longer training run first, so it's a worthier opponent. + +## 6. Risks + +1. **Browser MCTS performance.** Chess move-gen per node is heavier than FruitCake's depth-3 search; hundreds of sims/move in transpiled TS may be slow. Mitigation: modest inference sims (100–300), a ply/time budget, and profile — it's a latency knob, not a correctness one. (Batched-leaf / WASM are later levers.) +2. **Polyglot language gaps.** The port may want a `switch`/`match`, enums, or a Math function not yet in `std.math`. Mitigation: design around them (i32 consts + `if/else`, flat arrays — proven by FruitCake) **or add them to `C:\Repos\MintPlayer.Polyglot` and PR** (now an available option). No nested-generic params. +3. **Re-validating the engine after the port.** Rewriting a perft-verified engine in another language risks a subtle regression. Mitigation: perft (25/25) + the encoding round-trip run against the *generated* engine as the M40.1 gate; a facade-vs-core parity test. +4. **Honest strength.** Still a small, briefly-trained net → beatable. Framed as a from-scratch, self-taught, learning AI (the charm), not an engine. + +## 7. Verification + +- **M40.1:** solution builds (the `.pg` transpiles into the assembly); **perft 25/25** + encoding round-trip on the generated engine; `SelfPlayCampaign` chess contract test still green. +- **M40.2:** C#-vs-generated `PgPolicyValueNet` parity within f32 tolerance on fixed positions; `chess_solver.ts` emitted + committed; the TS parser round-trips a shipped `.ckpt` (a small Node/Jest or a Playwright check). +- **M40.3:** in-browser, a full legal game vs the AI, **no `/api/chess/*` move calls** (Network panel/Playwright), check/mate/draw detected, last-move highlight. Honest latency (~1–2 s/move at the chosen sims). + +See [PLAN.md](PLAN.md) M40. + +## 8. Reference appendix (execution details — read this if picking up cold) + +### 8a. What M39 already shipped (branch `m39-chess-selfplay-plan`, PR #32, stacked on the M38 branch #31) +The chess self-play stack is **done and committed**; M40 builds on it. Key artifacts: +- **Reusable seam/search/training (Core + Lab):** `Core/Planning/IZeroSumGame.cs` (+ `GameResult`), `Core/Planning/Mcts.cs` + (PUCT: select `Q+U` → expand-with-priors → net-leaf → **value negated every ply** → `Search` returns the root + visit-count π; Dirichlet root noise + `Config(Simulations, Cpuct=1.25, DirichletAlpha=0.3, RootNoiseFrac=0.25)`), + `Lab/PolicyValueTraining.TrainStep` (soft-CE(π) + MSE(`tanh` value, z)), `Lab/SelfPlayCampaign` + (`ITrainingCampaign`; plays MCTS self-play → `(obs,π,z)` window → trains `PolicyValueNet`; `--opponent-random` frac + mixes learner-vs-random games; store ids below). +- **Chess (Environments/Chess/):** `ChessBoard.cs` (`ChessState` + `ChessRules`: `LegalMoves`, `MakeMove`, + `IsSquareAttacked`, `InCheck`, `Result`, `Perft`; mailbox `sbyte[64]`, sq = rank*8+file, White moves +rank; piece + codes ±1..6 = P,N,B,R,Q,K; castling byte bits WK=1,WQ=2,BK=4,BQ=8; **threefold repetition NOT modelled** — 50-move + + ply cap bound loops), `ChessFen.cs`, `ChessMoveEncoding.cs` (AlphaZero **4672 = 64×73**: 56 queen planes [8 dir ×7 + dist] + 8 knight + 9 underpromotion; queen-promo rides the queen planes, decoded promo inferred on apply), + `ChessGame.cs` (`IZeroSumGame`; `PolicySize=4672`, `ObservationSize=1152` = 18 planes×64: [0–5] White + P,N,B,R,Q,K, [6–11] Black, [12] side-to-move, [13–16] castling WK/WQ/BK/BQ, [17] en-passant square). +- **Lab:** `--game chess` (`ChessLab.cs`, flags `--sims --games --eval-games --hidden --opponent-random`), + `--game chess --demo` (`ChessDemo.cs`: net as White + MCTS vs random Black, prints one FEN per ply to stdout), + `--game connect4` (`Connect4Lab.cs` + `Environments/Connect4/`). +- **Net:** the shared `Core/Nn/PolicyValueNet.cs` (variable-depth ReLU trunk → policy logits + scalar value), used + directly (no chess wrapper). **Value head is linear** — self-play wraps it in `tanh` at the call site. +- **Tests:** `ChessPerftTests` (25/25, incl. startpos d5=4,865,609, Kiwipete d4=4,085,603), `ChessEncodingTests` + (encode→decode→apply round-trip + mate/stalemate), `Connect4Tests` (MCTS-vs-negamax), `SelfPlayCampaignTests` + (resume roundtrip + vs-random). 355 fast + deep-perft (Slow). + +### 8b. Model-store ids / checkpoint (what the browser must parse) +`SelfPlayCampaign` saves to store `(env, algo)`: net = `("chess","az")` written `PolicyValueNet.Save(stream, kind:"selfplay-pv")`; +Adam = `("chess","az-adam")`. File on disk: `/chess.az.ckpt`. **The TS `.ckpt` parser (chess-net.ts) reads +(little-endian, mirroring `CheckpointFormat` + `PolicyValueNet.Save`; see `fruitcake-net.ts` for the exact idiom):** +``` +uint32 magic = 0x434E4C52 ("RLNC") +string kind = "selfplay-pv" (BinaryWriter 7-bit-encoded-int length prefix + UTF-8 bytes) +int32 version (= 2) +int32 trunkCount, then trunkCount × int32 (the hidden widths, e.g. 256,256) +per layer, in Parameters() order = [each trunk layer, then policyHead, then valueHead]: + int32 wCount + wCount × float32 (weight, row-major [inDim, outDim] → w[i*outDim + o]) + int32 bCount + bCount × float32 (bias) +``` +`inputSize` (1152) and `actions` (4672) are **not** stored — supply them from `ChessGame` (they’re Load params). +Forward: `x = obs; for each trunk layer: x = relu(x·W + b); logits = x·W_pol + b_pol; value = tanh(x·W_val + b_val)`. +MCTS priors = masked-softmax of `logits` over legal moves; leaf value = `value`. + +### 8c. Commands +``` +# train chess (writes /chess.az.ckpt; time-bounded; --opponent-random for robustness): +dotnet run -c Release --project tools/MintPlayer.AI.ReinforcementLearning.Lab -- \ + --game chess --hours 2 --sims 48 --games 10 --opponent-random 0.25 --data --seed 7 +# watch a game (prints FENs; loads the checkpoint from ): +dotnet run -c Release --project tools/…Lab -- --game chess --demo --sims 400 --demo-plies 120 --data --seed 11 +# perft/encoding gate: dotnet test --filter "FullyQualifiedName~ChessPerftTests" (add Category!=Slow for shallow) +# Polyglot transpile (bundled CLI; win-x64): +# ~/.nuget/packages/mintplayer.polyglot.msbuild/0.3.1/tools/win-x64/polyglot.exe \ +# build .pg --target typescript --out (and --target csharp) +``` + +### 8d. Polyglot facts (from the FruitCake precedent — `Environments/FruitCake/polyglot/`) +- Source: `fruitcake_solver.pg` (single source) → C# in `obj/` (build-time, MSBuild PackageReference + `MintPlayer.Polyglot.MSBuild` v0.3.1, `PrivateAssets=all`) + committed TS `fruitcake_solver.ts`. macOS dev must + point `$(PolyglotTool)` at a local `polyglot` binary. `dotnet watch` re-transpiles on save. +- Syntax: `import { List } from "std.collections"`, `import { Math } from "std.math"`; `record X(f: t)`; `class X { var f: t; init(...) {...}; fn m(...): t {...} }`; `fn f(a: i32): t => expr`; `for i in 0..n`, `for v in list`; + types `f64 i32 bool List T?`; `Math.min/max/sqrt/exp/tanh/log/PI`. Generated C# types are `Pg`-prefixed + internal, + wrapped by a hand-written **facade** (`FruitCakeWorld.cs`); a **parity test** pins facade↔core. +- **Constraints:** no nested-generic params (`List>` mishandled → flat `List` + offsets, see + `PgDuelingNet`); prefer `i32` consts + `if/else` over enums/`switch`. If blocked, extend Polyglot at + `C:\Repos\MintPlayer.Polyglot` and PR (owner-authorized 2026-07-12). +- Browser wiring (analogs to build): `chess-net.ts` (`.ckpt` parser → `PgPolicyValueNet`), `chess-director.ts` + (runs transpiled mcts+net), the Angular `chess/` component + route; ship `wwwroot/models/chess.az.ckpt` (LFS) + the + `.ckpt`→`application/octet-stream` static-file mapping in `Program.cs` (see how FruitCake's net is served). + +### 8e. Status of the "watch" demo (already delivered) +A browser **replay** artifact of one AI game exists (published via the Artifact tool from `--game chess --demo` +FENs). That is watch-only; M40 is the interactive *play-against-it* page. The demo net was trained ~21 min on CPU +(reached ~50% vs random, policy loss 4.4→2.2) — legal but weak; longer training recommended before M40.3. + +## 9. Difficulty (M40.4) — design from the investigation team (2026-07-13) + +**Goal:** let the visitor pick a difficulty in **both** modes (Play the AI, Watch AI-vs-AI). Three read-only agents +investigated the training/checkpoint machinery, the web surfaces, and the difficulty-composition design; this section +is the synthesis. + +### 9.1 What difficulty is made of — decision + +Compose difficulty from three possible axes, in priority order: + +1. **Search budget (`sims`) on ONE net — the spine.** Already a live knob (`ChessDirector.sims` → `PgChessMcts.chooseMove(net,state,sims,cpuct)`). More sims = stronger, slower; it's the classic engine difficulty lever (search depth/time), monotonic and reproducible, bounded by the in-browser latency budget (~1–2 s/move). Zero download cost, zero `.pg` change. +2. **Move-selection temperature — for human-like *variety* at the low end.** `PgChessMcts.search` already returns the full **visit-count distribution π** over 4672; `chooseMove` is just its argmax. So temperature lives **caller-side in `chess-director.ts`**: sample `∝ πᵢ^(1/T)` over the **visited** moves (every candidate is one MCTS actually explored → varied but never "unexplored garbage"). `T=0` = today's argmax. **No `.pg` change and no RNG in the Polyglot core** (`Math.random()` is browser-only; the single source stays pure/deterministic). Optional stronger variety lever: top-k-visited sampling. +3. **Network ladder (different-strength checkpoints) — optional NOVELTY only.** The owner's idea. Verdict: *not* the difficulty spine. This is a small, briefly-trained MLP — intermediate checkpoints are noisy and **not reliably rankable** (an under-trained net has a near-uniform policy → plays *confused*, i.e. the "randomly dumb" feel we want to avoid at Easy), and each net is **~5–6 MB** LFS. Its real value is narrative: *"play the AI at an early stage of its self-taught learning."* Ship **at most two** nets (a "Rookie (early training)" novelty + the final), framed as a novelty, not a balanced tier. + +**Why Easy should weaken via fewer sims (+ mild temperature), NOT a weaker net or high temperature:** fewer sims still plays the net's *considered* move (decent priors) — it loses *gracefully* (doesn't calculate deep tactics) rather than hanging pieces at random. A weaker net or high-T sampling both feel "randomly dumb." Floor: keep Easy at **low-but-nonzero** sims (~24) — `sims=0` returns raw priors (the `total==0` fallback) and loses the one-ply tactical safety net. At very low sims, visits concentrate, so temperature has little to sample among → Easy needs the *combination* (low-nonzero sims + slightly higher `cpuct` + `T≈1`). + +**Honest scope / labels:** the strongest shippable config (best net, ~256–300 sims, `T=0`) is still club-beatable. Label the top tier **"Full strength"** (or "Hard") — **never "Grandmaster."** A longer training run lifts the whole ladder uniformly but doesn't change its composition. + +### 9.2 Proposed ladder + +| Tier | Checkpoint | sims | temperature | cpuct | Feel | +|---|---|---|---|---|---| +| Easy | final net | ~24 | ~1.0 | ~2.0 | fast (<0.5 s), varied, casual — misses tactics | +| Medium | final net | ~96 (current) | ~0.5 | 1.5 | ~1 s, mostly sensible, occasional miss | +| Full strength | final net | ~256–300 | 0 (argmax = today) | 1.25–1.5 | ~1–2 s, the net's genuine best | +| *Rookie (opt.)* | *early checkpoint* | ~64 | ~0.6 | 1.5 | *novelty: "watch it before it learned"* | + +### 9.3 Delivery — a committed manifest (shippable now on ONE net) + +Ship `wwwroot/models/chess-difficulties.json` (static; `.json` is already served — mirrors the `rushhour-deck.json` precedent), fetched by the director at load, with a hardcoded fallback: + +```json +[ + { "label": "Easy", "ckpt": "/models/chess.az.ckpt", "sims": 24, "temperature": 1.0, "cpuct": 2.0 }, + { "label": "Medium", "ckpt": "/models/chess.az.ckpt", "sims": 96, "temperature": 0.5, "cpuct": 1.5 }, + { "label": "Full strength", "ckpt": "/models/chess.az.ckpt", "sims": 256, "temperature": 0.0, "cpuct": 1.5 } +] +``` + +**First cut needs only the one net we already ship** — the three tiers differ by `sims`/`temperature`/`cpuct`. Adding real multi-checkpoint tiers later is a manifest edit + dropping `.ckpt` files into `wwwroot/models/` — **zero code change** (`chess-net.ts` `loadChessNet(url)` is already parameterized and reads trunk widths from the file). Missing-ckpt caveat: the SPA fallback returns `index.html` with `resp.ok===true`, but the parser throws on the magic check → `catch` → `null` → the director's random-move fallback (silent degradation — surface it in `statusText`). + +### 9.4 Code changes + +- **`chess-director.ts`:** add `difficulties[]` + `current` (loaded from the manifest, hardcoded fallback); `setDifficulty(d)` sets `sims`/`cpuct`/`temperature` and re-fetches the net **only if the ckpt URL changed** (cache nets by URL, so sims-only switches don't refetch); in `aiStep()`, use `PgChessMcts.search(...)` + temperature sampling (`T=0` → keep `chooseMove` argmax). +- **`chess.ts`:** a difficulty selector (segmented control / dropdown) in **both** modes. Play = the opponent's strength. Watch = a **single shared level first**; per-side White/Black levels (the "strong vs weak" demo) is a clean follow-up (director holds two nets + picks by `whiteToMove`). Changing difficulty in watch mode should `loopGen++` + `reset()` (mirror `setMode`) and re-await readiness. +- **No change** to `chess-net.ts`, `Program.cs`, `CheckpointFormat`, `PolicyValueNet`, `chess_solver.pg`, or `.gitattributes`. + +### 9.5 Capturing a net ladder (only if we do the Rookie/novelty or per-side tiers) + +Training overwrites `chess.az.ckpt` in place ~every 10 min (`CampaignRunner` eval cadence); no history is kept, and the only strength signal is **winRate-vs-random** (saturating, noisy at 10 eval games) + games-trained — **there is no net-vs-net Elo/arena**. To capture a ladder: +- **Option B (zero code, recommended for a 1–2-net novelty):** manually copy `/chess.az.ckpt` to a tier name (`chess.az.l1.ckpt`) at a chosen milestone. +- **Option A (automated):** a `--ladder-winrates`/`--ladder-games` flag in `ChessLab` → `SelfPlayCampaign.Checkpoint` dumps named snapshots (`store.Save(env, "az-tier{n}", …)`; the store already namespaces by algo id, so no store change). +- A **net-vs-net arena** (an earlier tier as opponent, paralleling `ArenaVsRandom`) is the one genuinely-new capability that would make tiers *reliably* ordered — deferred; not needed for sims+temperature difficulty. + +### 9.6 Refinement (2026-07-13, owner) — auto-captured net ladder, produced by the Lab (SUPERSEDES the "novelty only" stance) + +Training is **always offline via the Lab** (the web app never trains; the Lab's `--viz` net visualizer is unrelated). The difficulty ladder should be produced **hands-off by the training agent**: when the live net becomes *significantly stronger than the last promoted difficulty checkpoint*, the Lab **auto-writes a new difficulty checkpoint straight into `src/RLDemo.Web/wwwroot/models/`** and updates the manifest — no manual copying. "Run `--game chess --ladder --hours N` and walk away; the web app's difficulty roster grows as the net improves." + +This makes the **net ladder the difficulty backbone** (superseding §9.1's "net-ladder = novelty only"): the promotion gate solves the "not reliably rankable" problem, because a checkpoint is promoted **only if it beats the current champion by a margin in a net-vs-net arena** (the capability §9.5 flagged as missing). Tiers are then reliably ordered by construction — each Level K+1 provably beats Level K. (Sims/temperature from §9.1–9.2 remain an *optional* per-tier layer on top; the net is now the primary axis.) + +**Mechanism** (in `SelfPlayCampaign` — generic, not chess-only; enabled by `--ladder`): +- **Champion** = the last promoted net, held as a *frozen snapshot* (save→reload, so continued training on the live net doesn't mutate it). Tier 1 = the first eval net (a weak baseline). +- On each eval/checkpoint, run **`ArenaVsNet(liveNet, champion, arenaGames)`**: full games to terminal; the side-to-move's net picks via deterministic MCTS (argmax, no root noise); games are diversified by a short **randomized opening** (a few random legal plies) drawn from a **dedicated arena RNG independent of the self-play/training RNG streams** (so training stays bitwise-reproducible — the M36 `--viz` invariant); colours alternate. Score = challenger wins + 0.5·draws, over `arenaGames`. +- If score ≥ `--promote-margin` (default ~0.58) → **promote**: save `chess.az.d{K}.ckpt` into `--difficulty-dir` (default `src/RLDemo.Web/wwwroot/models`), rewrite `chess-difficulties.json` with all K tiers, set champion = frozen copy of the live net, `K++`. (Guard against thrashing: promote at most once per eval; require the margin over a full arena.) +- **Manifest** entry per tier: `{ "label": "Level K", "ckpt": "/models/chess.az.d{K}.ckpt", "sims": …, "temperature": …, "cpuct": …, "winRateVsRandom": … }`. The **net** is the difficulty axis; sims/temperature carry §9.1 defaults (optionally weaker/among lower tiers). +- **Flags** (`ChessLab`): `--ladder` (enable), `--difficulty-dir ` (default the web models dir), `--promote-margin <0..1>`, `--arena-games `. + +**Reproducibility:** the arena is inference-only (no weight updates) on a **separate RNG stream** (`_arenaRng`, seeded independently of the self-play/eval streams), so it does not alter the training *trajectory* — the same seed yields the same sequence of nets. It is *not* free in wall-clock, though: because runs are time-bounded (`--hours`), the arena's per-eval cost means a `--ladder` run completes marginally fewer training games in the same wall-clock than a non-ladder run (it spends some time evaluating). So "same seed ⇒ same net after N training games" holds; "same net after H hours" does not. + +**Promotion gate — two signals, OR'd** (learned during the build): a pure net-vs-net score can't rank *weak* nets, because two nets that can't force checkmate **draw** each other (ply-cap) → ~50% regardless of strength. So promote when EITHER (a) **winRate-vs-random rose by ≥ `--promote-margin`** over the champion's recorded value (the discriminating signal while weak), OR (b) the **head-to-head arena score ≥ `--arena-margin`** (the signal once winRate-vs-random saturates near 100% and nets can actually convert wins). Verified: a forced run produced an ordered 5-tier ladder (`chess.az.d1…d5.ckpt` + manifest); the gate correctly declines when neither signal is met. + +**Files changed (M40.4a, done):** `SelfPlayCampaign.cs` (`ArenaVsNet`/`ModelMoveWith`/`EvaluateWith`, champion state, `MaybePromoteDifficulty`/`PromoteTier`, JSON manifest writer, ladder-resume), `ChessLab.cs` (`--ladder`/`--difficulty-dir`/`--promote-margin`/`--arena-margin`/`--arena-games`/`--difficulty-sims`/`--opening-plies` + configurable `--first-eval`/`--eval-every`), `LabHost.cs` (optional eval-cadence params). No web-serving/`.gitattributes`/`CheckpointFormat` change. diff --git a/docs/prd/PARALLEL_SELFPLAY_PRD.md b/docs/prd/PARALLEL_SELFPLAY_PRD.md new file mode 100644 index 0000000..403a2f5 --- /dev/null +++ b/docs/prd/PARALLEL_SELFPLAY_PRD.md @@ -0,0 +1,98 @@ +# Reusable deterministic CPU-parallel data generation (self-play + cube) — PRD + +**Status:** ✅ SHIPPED 2026-07-13 (branch `m39-chess-selfplay-plan`, PR #32) — all of M41.1/M41.2/M41.3. +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M41 · **Motivated by:** chess self-play is CPU-bound (movegen) and single-threaded, so a multi-core box sits ~idle while training crawls (see the M40.4 material-shaping run: 256 sims/move, one core). + +## Implementation status (what actually shipped vs. this design) +- **M41.1** `Core/Training/DeterministicParallel.cs` (commit `bc0c48f`). Two overloads: the `SeedSequence`+stream form in + §4a (with `int stream`, not `uint`), plus a **raw-`ulong baseSeed` overload** added in M41.3 (the SeedSequence form + delegates to it). 12+1 unit tests: bitwise parallel==sequential across DOP 1/2/4/8/16, ordering, distinct streams, + edges, and the cube-seed-equivalence lock. +- **M41.2** parallel self-play (commit `a9fa5c3`). Per §4b, minus: **eval/arena were left sequential** (not the + bottleneck, and they can't affect trained weights), and no separate arena RNG work was needed. Gate MET — Connect-4 + **byte-identical checkpoint** at sequential vs dop-1 vs dop-8. +- **M41.3** cube dedup (commit `bed7577`) — **done, not skipped.** Better than this doc's "outputs may change, re-verify" + assumption: the raw-seed overload reproduces the old `roundBase + φ·(worker+1)` seeding **byte-for-byte**, so cube + output is unchanged (locked by a test). The shared `Interlocked` solve-counter was removed (per-generator counts summed + on the owner thread). +- **Not done:** a formal games/hour speedup measurement on chess (the DOP-invariance gate + architecture make the speedup + structural; left to observe in a real training run). + +## 1. Problem + +AlphaZero self-play (`SelfPlayCampaign`, used by chess/connect4) generates its training games on **one thread** — `TrainChunk` runs `for (g…) PlayGame()` serially (`SelfPlayCampaign.cs:119`). For chess, wall-time is dominated by **legal-move generation inside MCTS** (every simulation calls `IZeroSumGame.LegalMoves`/`Apply` → `ChessGame`/`ChessRules`), which is embarrassingly parallel per game — yet all but one CPU core sits idle. This is the practical bottleneck behind "training is too slow" (more so than the GPU angle: the net is tiny and inference is batch-1, a poor GPU fit). + +Meanwhile the repo **already parallelizes data generation for the Rubik's cube** — but that code is **hand-rolled and duplicated in two game-specific Lab files**, not in the reusable library. The owner's question: *why isn't this in Core, and should we refactor it so self-play (and everything else) gets it for free?* + +## 2. Findings (from a 2-agent read-only investigation, 2026-07-13) + +### 2a. Where CPU parallelism lives today +- **Game-specific, in the Lab (duplicated):** cube training-data generation — + `CubeImitationCampaign.cs:71` (Kociemba-oracle self-play) and `CubeEfficientCampaign.cs:93` (teacher-free scramble-reversal). Both hand-roll the **same** idiom: `Parallel.For(0, ProcessorCount-2, …)` + **per-worker result lists** (disjoint indices) + **per-worker seeded `Xoshiro256StarStar`** + an `Interlocked` counter. Copy-pasted between the two; **not in Core**. +- **Reusable, in Core (generic):** `ManagedBackend` GEMM/row-ops (`IComputeBackend.cs:426`, MAC-thresholded, disjoint row bands — the NN math under *every* campaign); `VectorEnv.Step` (`VectorEnv.cs:67`, per-env RNG, disjoint slots, `parallel:true`); `ValueIterationTrainer` batch featurization (`ValueIterationTrainer.cs:188/218`, used by `cube-davi`). +- **No explicit parallelism:** `SelfPlayCampaign` and the DQN campaigns (`SnakeDqnCampaign`, `FruitCakeDqnCampaign`, `DqnScoreCampaign`) — their only CPU parallelism is the implicit backend GEMM. + +**So: there is no reusable "parallel data-generation / episode rollout" primitive in Core.** Each Lab campaign that wants it hand-rolls the pattern; self-play never did. There is **no deliberate reason** — it's organic growth, and the hand-rolled idiom is *exactly* the deterministic pattern Core already uses elsewhere (`VectorEnv`, GEMM). It should be extracted. + +### 2b. Is self-play safe to parallelize? Yes — with a per-game-RNG + ordered-merge. +- **Shared read-only net inference is SAFE concurrently.** `PolicyValueNet.Forward` → `Tensor` ops allocate fresh output buffers, never mutate inputs, and there is no shared scratch/static cache (`TensorOps.cs:16/56`, `IComputeBackend.cs:189`). The one caveat is the autograd tape: weights are `RequiresGrad` (`Modules.cs:53`), so a forward *outside* `NoGrad` would build a tape over shared weights — but the evaluator already wraps every forward in `using (GradMode.NoGrad())` (`SelfPlayCampaign.cs:286`), and `GradMode` depth is **`[ThreadStatic]`** (`Tensor.cs:143`), entered per worker thread. Batch-1 forwards also don't trip `ManagedBackend`'s `rows>=2` gate, so they won't spawn nested `Parallel.For` fighting the outer per-game loop. → **one shared read-only net snapshot suffices; no per-thread net copies.** +- **The true blockers** are (i) the shared `_window` (`List.Add`/`RemoveAt`) + non-atomic `_totalGames`/`_totalSamples` counters, and (ii) the shared mutable RNGs (`_searchRng`/`_evalRng`/`_arenaRng`) — a single advancing Xoshiro shared across concurrent games both **races** and **breaks determinism** (a game's draws depend on interleaving). +- **Bottleneck** is chess `IZeroSumGame` movegen inside MCTS (heavy, per-simulation), NOT net inference — which is exactly why per-game thread parallelism pays off and a shared net is fine. + +### 2c. The reproducibility invariant — must be preserved +The repo guarantees **bitwise determinism per seed** (PLAN M25/M26; **M36 SHA256-verified** viz-vs-no-viz). Core's existing parallel primitives keep it by construction: GEMM partitions **disjoint output rows, no reduction → byte-identical at any DOP** (`IComputeBackend.cs:115`); `VectorEnv` gives **each env its own RNG → parallel == sequential** (`VectorEnv.cs:9`). The refactor MUST match this: a game's randomness and output slot must be a pure function of its **global game index**, not execution order. + +## 3. Decision + +**Yes, refactor.** Extract a small **reusable, deterministic CPU-parallel sample-generator into Core**, and adopt it in `SelfPlayCampaign` (new capability: parallel self-play) and the two cube Lab campaigns (dedup). This gives self-play ~N-core scaling on its chess-movegen bottleneck **while keeping bitwise reproducibility**, and removes the copy-pasted cube idiom. + +## 4. Design + +### 4a. The Core primitive (`Core/Training/` or `Core/Concurrency/`) +A generic, determinism-preserving parallel generator — the common shape of both cube data-gen and self-play: + +```csharp +public static class DeterministicParallel +{ + /// Runs `makeItem(index, rng)` for index in [0,count), each with its OWN RNG derived from + /// (seeds, stream, baseIndex+index) — so a given index yields identical output regardless of + /// worker count or completion order. Results are returned in ASCENDING index order. `parallel` + /// toggles Parallel.For vs a sequential loop; the two are bitwise-identical (VectorEnv/GEMM rule). + public static TItem[] Generate( + int count, SeedSequence seeds, uint stream, long baseIndex, + Func makeItem, bool parallel, int? maxDop = null); +} +``` + +- **Per-item RNG:** derive as `VectorEnv`/`SeedSequence` already do (`baseSeed + index*0x9E3779B97F4A7C15UL`, `VectorEnv.cs:54`; `SeedSequence.Derive`) so item *g* is reproducible independent of scheduling. +- **Ordered result slots:** write `results[index]` (preallocated), exactly like `VecStep`'s per-index arrays — the parallel region touches only disjoint slots (no locks), and the caller consumes them in index order. +- **Bitwise guarantee** identical to `ManagedBackend(maxDegreeOfParallelism)`: the DOP/`--parallel` knob provably does not change output. + +### 4b. Self-play adoption (`SelfPlayCampaign`) +- Move the single-game logic (`PlaySelfPlay`/`PlayVsRandom`) into a **pure, index-addressable** form that takes its own `Xoshiro` (for the mode coin-flip, MCTS Dirichlet, and `SelectMove`) and a **shared read-only net snapshot** (`_net`, or a `Freeze(_net)` to decouple from an in-flight train step), and **returns** its samples instead of calling `AddSample`. +- `TrainChunk` calls `DeterministicParallel.Generate(gamesPerChunk, …, makeItem: g => PlayOneGame(g, …), parallel)` → then **merges the returned per-game sample lists into `_window` in ascending game index** and only then bumps `_totalGames`/`_totalSamples`. Training (`PolicyValueTraining.TrainStep`, `Backward`, optimizer) stays on the **owner thread** after the join. +- The eval/arena loops (`ArenaVsRandom`, `ArenaVsNet`) are structurally the same independent-game loops → same primitive, per-game eval/arena RNG. +- A `--parallel`/`--dop` flag (default = cores−2, matching cube; 1 = today's behaviour). **Weights are identical at any DOP** (assert via a SHA check like M36). + +### 4c. Cube dedup (secondary) +`CubeImitationCampaign`/`CubeEfficientCampaign` re-express their `Parallel.For + per-worker-list + seeded-RNG` block as `DeterministicParallel.Generate(...)`, removing the duplication. (Their `makeItem` calls `CubeOracle.LabelScramblePath` / `CubeSelfSupervised.LabelScramblePath` — the primitive is general enough; it's just "produce K labeled samples per index".) + +## 5. Phases + +- **M41.1 — the Core primitive + tests.** `DeterministicParallel.Generate` + unit tests proving parallel output == sequential output (bitwise), for several DOPs, on a toy `makeItem`. +- **M41.2 — parallel self-play.** Refactor `SelfPlayCampaign` generation onto it (pure per-game fn + per-game RNG + ordered merge; shared read-only net); `--parallel`/`--dop`. **Gate:** SHA256-identical trained checkpoint at dop-1 vs dop-N for a fixed seed/short run (the M36-style determinism proof), self-play contract tests green, and a wall-clock speedup on chess (report games/hour at dop-1 vs dop-N). +- **M41.3 (optional) — cube dedup.** Move the two cube campaigns onto the primitive; verify their outputs unchanged (SHA) and DOP-invariant. + +## 6. Risks +1. **Reproducibility regression** — the whole point; mitigated by the per-index-RNG + ordered-merge design and a SHA dop-1-vs-N gate (M41.2). +2. **Hidden shared state in inference** — investigation found none (fresh buffers, `[ThreadStatic]` NoGrad); guard by keeping `Backward`/optimizer strictly off worker threads. +3. **Nested parallelism** (outer per-game × inner GEMM) — batch-1 self-play forwards don't trip the GEMM `rows>=2` gate, so no oversubscription; if it ever does, pin the inner backend to DOP-1 during generation (the `FruitCakeAb` precedent, `FruitCakeAb.cs:36`). +4. **Scope creep** — DQN campaigns could also adopt it later; out of scope for M41 (they're not the bottleneck). + +## 7. Verification +- M41.1: parallel==sequential bitwise unit test across DOPs. +- M41.2: **SHA256(checkpoint) identical dop-1 vs dop-8** for a fixed seed + short run; `SelfPlayCampaignTests` green; measured games/hour speedup on `--game chess`. +- M41.3: cube checkpoints unchanged + DOP-invariant. + +See [PLAN.md](PLAN.md) M41. diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index d0bb485..8b3feba 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1359,6 +1359,221 @@ shallow/leaky, PR #27's correct call), the 3 imitation/policy campaigns *as whol policy campaign SHA256-bitwise-identical vs baseline; `--viz` live for every game; net **negative** line diff with every shared module carrying an interface comment. See `BOILERPLATE_REDUCTION_PRD.md`. +## M39 — Self-play training (Connect-4 → chess) *(2026-07-12; branch `m39-chess-selfplay-plan`, PR #32; see `CHESS_SELFPLAY_PRD.md`)* — M39.1 + M39.2 SHIPPED + +**Status (2026-07-12):** **M39.1 (rails on Connect-4) and M39.2 (chess) SHIPPED**, each its own commit on +`m39-chess-selfplay-plan`. The reusable stack — `IZeroSumGame` + `Core/Planning/Mcts.cs` (PUCT) + +`PolicyValueTraining` + `SelfPlayCampaign` — is in Core/Lab; Connect-4 and chess are both consumers, the +latter reusing the rails **unchanged**. Chess movegen is **perft-verified** (25/25 published counts, incl. startpos +depth 5 and Kiwipete depth 4); the 4672 move encoding round-trips (encode→decode→apply) with no collisions; a +`--game chess` run from random init plays legal chess and beats a random-legal opponent. A **robustness slice of +M39.3** also shipped: `--opponent-random` mixes in learner-vs-random games so the net trains on the off-distribution +positions an unexpected move reaches (the direct code answer to "a novel move disorients the AI"). 361 fast tests +green (perft, MCTS-vs-negamax, encoding round-trip, both self-play contracts). The rest of M39.3 (batched-leaf MCTS, +GPU/conv, league play, web page) remains optional future work. Honest scope holds: legal, steadily-improving play, +not engine strength. + +**Problem.** The SDK trains via reward (DQN/PPO), a forward model (DAVI), or an exact oracle (cube/Rush Hour imitation). Chess +has no cheap oracle and a reactive net plateaus — the repo's own history says **search is the lever**. The missing paradigm is +**self-play**: the improving net plays itself and the games *are* the training signal (AlphaZero-style). Add it as a **reusable** +capability — chess is the headline, but the machinery is shared so the next two-player game plugs in by writing only its rules. + +**Key decision (design-it-twice, 3-agent investigation 2026-07-12).** AlphaZero-style (MCTS-guided self-play + the two-headed +`PolicyValueNet`), **not** plain reactive self-play (tactically blind — the documented plateau). The chess rules engine is the +same irreducible cost either way, so the marginal cost of "real improvement" is just MCTS + visit-count targets. One new **deep** +seam — `IZeroSumGame` in `Core/Planning` (a *sibling* to `IDeterministicModel`: side-to-move + win/loss/draw + per-state +legal moves, which `IDeterministicModel` can't honestly express) — consumed by both MCTS and the self-play campaign. **Reuse-first:** +~70% of the outer loop already exists (`PolicyValueNet` unchanged, the soft-CE+value train step from the imitation campaigns, +`ITrainingCampaign`/`CampaignRunner`, model store + checkpoint format, the M38 `AdamState`/`TrainWindow`/`PolicyGrowth` plumbing, +action masking, RNG streams, `--viz`, the A/B harness → Elo gate). New code is confined to the seam, MCTS, the `(obs,π,z)` +self-play loop, and each game's rules. + +**Phased plan (each step ends on a green build + its gate; ordered to de-risk the *novel* machinery before the huge rules surface).** + +- **M39.1 — the rails, proven on Connect-4.** `IZeroSumGame` + `Core/Planning/Mcts.cs` (PUCT, sign-flipping value backup, + Dirichlet root noise, temperature → root visit-count π) + `PolicyValueTraining.TrainStep` (generalized from + `CubePolicyTraining.TrainStep`: soft-CE + value regression) + `SelfPlayCampaign : ITrainingCampaign` (reusing + `AdamState`/`TrainWindow`/`PolicyGrowth`/telemetry) + `Connect4Game : IZeroSumGame` + a **negamax** oracle for tests. **Gate:** + MCTS unit-tested vs negamax on forced-win/draw positions; from random init, self-play win-rate vs negamax/random **climbs**; + `CampaignContractTests` resume roundtrip; fast suite green. This validates the self-play machinery cheaply, away from chess rules. +- **M39.2 — chess as consumer #2.** `ChessBoard` + full legal movegen + draw/mate detection; the 8×8×73 = 4672 move encoding; + `ChessGame`/`ChessEnv` (`IEnvironment` + `IActionMaskProvider` + `IStatefulEnvironment`); flattened-plane observation (static, + train==serve); a thin `ChessPolicyNet` wrapper over `PolicyValueNet` (policy head 4672, value `tanh` → WDL); `ChessSelfPlayCampaign` + + `--game chess` Lab dispatch; an Elo eval (`ChessAb`, mirroring `FruitCakeAb`). **Gate (the hard one): perft** node-counts matched + to published values (startpos, Kiwipete, …) to depth 5–6 *before any training*; move encode/decode round-trips; terminated-vs-truncated + split correct; win-rate vs random-legal climbs; contract-test resume; ship `models/chess.az.ckpt` (LFS). +- **M39.3 — scale + robustness (optional).** Batched-leaf MCTS; GPU-resident forward *if* the net grows enough to clear the + routing threshold (honest: a small chess MLP won't); conv-backend support for positional strength (a separate backend workstream); + a web showcase page. **Anti-exploitation levers** (so a novel/weak human move can't disorient the net into losing — real even at + superhuman level, cf. the KataGo cyclic-group exploit): opponent-**pool/league** play + an occasional random/weak mover, diverse + randomized opening positions, and adversarial fine-tune on any discovered exploit. (Dirichlet root noise + temperature already ship + in the M39.1 MCTS/campaign; search-from-the-actual-position is the primary defence. NoisyNets is *not* the right tool — it perturbs + the policy globally rather than broadening position coverage.) + +**Honest scope.** MLP-only net (no conv) over a flattened board → *legal, steadily-improving* play, not engine strength; self-play is +CPU-bound (the small MLP won't hit the GPU lever that helps the cube). Connect-4 is where the self-improvement curve is unmistakable; +chess is the headline consumer. **Left out of v1:** DQN/PPO for self-play (wrong fit), the DQN `ReplayBuffer` (uses an `(obs,π,z)` +window instead), bitboards (unless a bench forces it), superhuman strength. **Whole-milestone gate:** builds 0/0; fast suite green; +perft passes; both games' self-play win-rate rises vs a fixed baseline; resume roundtrips. See `CHESS_SELFPLAY_PRD.md`. + +## M40 — Play the chess AI in the browser (single-source via MintPlayer.Polyglot) *(2026-07-12; see `CHESS_WEB_POLYGLOT_PRD.md`)* — M40.1–M40.4 ✅ SHIPPED (net committed; conv-net strength upgrade tracked in M42) + +**Goal.** Play the self-taught chess AI (M39) **in the browser**, client-side, with **zero server inference** — the +FruitCake pattern (ARCHITECTURE §10): write the inference path once in a `.pg`, transpile to C# (training/serving) + +TypeScript (browser); the browser downloads and parses the `.ckpt` and runs the net + MCTS locally. The wrong path +is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); the right path is single-source. + +**Feasibility — VERIFIED (2026-07-12)** by transpiling a probe with the bundled `polyglot.exe`: +- `Math.exp`, `Math.tanh`, `Math.log`, `Math.sqrt` all transpile (need `import { Math } from "std.math"`), so MCTS's + masked-softmax priors + `tanh` value + PUCT `sqrt` are expressible. Transcendentals are **not** bit-exact across + C#/JS (only `+ - * / sqrt` are — the reason FruitCake avoided them) but chess **inference doesn't need bit-exactness** + (the browser AI must play well, not match C# to the ULP), so `exp`/`tanh` are fine. +- Bitwise ops `& | << >>` transpile (castling flags; booleans also fine). +- **Polyglot itself is extendable:** the toolchain lives at `C:\Repos\MintPlayer.Polyglot` — a missing feature + (`switch`/enum/Math fn) can be **added there and PR'd** (owner-authorized). Known limits to design around + (from the FruitCake solver): **no nested-generic params** (use flat `List` + offsets like `PgDuelingNet`), + prefer `i32` consts + `if/else` over enums/`switch`. + +**Phases (each ends on a green build + its gate).** +- **M40.1 — single-source the engine. ✅ SHIPPED 2026-07-12.** Ported `ChessBoard`/`ChessRules` + `ChessMoveEncoding` + into `Environments/Chess/polyglot/chess_solver.pg` (internal `Pg`-prefixed core: `PgChessState` + `PgChessMove` — + board `List[64]`, promotion/castling as `i32`, bounded `for`+`continue` rays, `List<(i32,i32)>` delta tables; + every construct proven by `fruitcake_solver.pg`). `MintPlayer.Polyglot.MSBuild` transpiles every `**/*.pg` to `obj/` + before CoreCompile (bumped 0.3.1 → **0.6.0**, which bundles win-x64 + linux-x64 + linux-arm64 + osx-x64 + osx-arm64, + so CI/ubuntu is unaffected and macOS dev no longer needs `$(PolyglotTool)`). `ChessState`/`ChessRules`/ + `ChessMoveEncoding` are now **thin C# facades** over the core (public API + both test files unchanged); `ChessGame`'s + seam (`LegalMoves`/`Apply`/`Result`) delegates to the core's `legalMoveIndices`/`applyIndex`/`result` so training and + the browser share one implementation; `perft` recurses entirely in-core. **Gate MET: perft 25/25 on the generated + engine** (incl. startpos d5 = 4,865,609, Kiwipete d4 = 4,085,603, ~10 s), encoding round-trip + terminal-detection + green, `SelfPlayCampaign` chess contract green, full suite 362/362 (no FruitCake/Snake/MountainCar regression from + the shared re-transpile). **Polyglot MSBuild multi-`.pg` incremental bug — FIXED upstream in 0.6.0 (PR #26):** a + single-`.pg` edit used to make MSBuild's partial-incremental build hand the CLI a subset → inline duplicate prelude + clash (CS0101/CS0260). Diagnosed here, verified a stamp-`Outputs` + `RemoveDir` fix against the source `.targets`, + and it shipped in 0.6.0; the temporary local `_PolyglotForceFullRetranspile` workaround has been removed + (`docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`). +- **M40.2 — single-source the inference math + a TS `.ckpt` parser. ✅ SHIPPED 2026-07-13.** Added to the `.pg`: + `writeObservation()` (18-plane × 64 = 1152, parity-tested vs `ChessGame.WriteObservation`), `PgPolicyValueNet.forward` + (flat-array ReLU trunk → policy logits + linear value), and `PgChessMcts` (inference PUCT — masked-softmax priors, + `sqrt`, value negated per ply, **no Dirichlet**). `ClientApp/src/app/chess/chess-net.ts` parses the `selfplay-pv` + `.ckpt` (magic RLNC, trunk widths, per-layer W/b in `Parameters()` order; inputSize/actions supplied) into the + generated `PgPolicyValueNet`; committed `chess_solver.ts` emitted. **Gate MET:** `ChessNetParityTests` — C# + `PolicyValueNet.Forward` vs the generated net agree within f32 tol on the start position (round-tripped through the + real `.ckpt` bytes), plus observation parity + an MCTS runtime smoke (valid legal-move distribution, `chooseMove` + legal); 358/358 fast tests. Also did the `std.math` cleanup (`Math.abs`/`Math.max`, kept `isign`). The two TS-emitter + gaps chess first hit (local `List` decls losing their annotation; `List` → `T | null[]`) were filed as + **MintPlayer.Polyglot issue #27** with a verified two-part fix, **fixed upstream in 0.7.0 (PR #28)**; the Environments + `.csproj` is on 0.7.0 and the regenerated `chess_solver.ts` is now **strictly typed** (`let d: [number,number][] = []`, + `children: (PgMctsNode | null)[]`) — verified strict-`tsc`-clean. 365/365 tests on 0.7.0. +- **M40.3 — the browser page. ✅ SHIPPED 2026-07-13 (net committed).** `chess-director.ts` (runs the + transpiled `PgChessMcts`+net over the loaded `.ckpt`) + a standalone Angular chess component: an 8×8 board (White at + the bottom, square rows), click-to-move validated by the transpiled engine (legal-target dots, orange + selected/last-move highlights, auto-queen), check/checkmate/stalemate/draw status, a **captured-pieces tray** (red ✕ + per taken piece), and **two modes — "Play the AI" and "Watch AI-vs-AI"** (self-restarting loop with a speed slider). + Route `/chess` + a Home tile; `.ckpt` MIME mapping already in `Program.cs`. **Gate MET:** Angular builds the chess + chunk clean; `/chess` served; `/api/chess/*` → 404 (zero server inference); director/net/solver strict-`tsc` clean; + and the transpiled engine+net+MCTS played a full 80-ply legal game in Node over the real checkpoint. Playwright MCP + was unavailable, so the in-browser click-through wasn't automated — verified structurally + functionally instead. + **Net SHIPPED (commit `1dd734d`):** `wwwroot/models/chess.az.d1.ckpt` (LFS) + `chess-difficulties.json` committed, so a + fresh deploy has a net to load; verified headless (loads + plays a full legal game). Honest caveat: this is the flat-MLP + net → legal-but-weak chess; the conv-net upgrade is tracked in **M42** (replaces it with no page changes). +- **M40.4 — difficulty via an auto-captured net ladder (both modes). ✅ SHIPPED (investigation + owner refinement + 2026-07-13; see `CHESS_WEB_POLYGLOT_PRD.md` §9, esp. §9.6).** Training is offline-only (the Lab); the ladder is + produced **hands-off by the training agent** — when the live net becomes *significantly stronger than the last + promoted checkpoint*, the Lab auto-writes a new difficulty `.ckpt` into `src/RLDemo.Web/wwwroot/models/` + updates a + manifest. The net ladder is the backbone (not a novelty) because promotion is gated on a **net-vs-net arena** margin, + making tiers reliably ordered by construction (Level K+1 provably beats Level K). + - **M40.4a — the Lab mechanism (the owner's ask).** In `SelfPlayCampaign` (generic; enabled by `--ladder`): + a champion = last-promoted frozen net; on each eval, `ArenaVsNet(liveNet, champion, arenaGames)` (deterministic + MCTS argmax, short randomized openings on a **separate arena RNG** → training stays bitwise-reproducible, + alternating colours); if challenger score ≥ `--promote-margin` (~0.58) → save `chess.az.d{K}.ckpt` to + `--difficulty-dir` (default the web models dir) + rewrite `chess-difficulties.json` (`{label, ckpt, sims, + temperature, cpuct, winRateVsRandom}`) + champion := frozen copy. Flags in `ChessLab`. **Gate:** a short ladder + run auto-produces ≥2 ordered tier ckpts + a valid manifest in `wwwroot/models`; a non-`--ladder` run's weights are + unchanged for the same seed (reproducibility check). + - **M40.4b — the web selector.** `chess-director.ts` loads `chess-difficulties.json` (hardcoded fallback), a + `setDifficulty(d)` (re-fetch net only if the ckpt URL changed; cache by URL; `aiStep` uses `search`+optional + temperature, `T=0`→argmax), and a Level-picker in both modes (Play = opponent; Watch = shared level, per-side is a + follow-up). No `.pg` change (`PgChessMcts.search` already returns π). Honest labels ("Level K" / "Full strength", + never "Grandmaster"). + +**Honest scope:** the M39 net is a small, briefly-CPU-trained MLP → legal, still-learning chess, beatable by a decent +human. **Non-goals:** engine strength, a server chess endpoint, bit-exact transcendentals, browser training. See +`CHESS_WEB_POLYGLOT_PRD.md` (incl. its reference appendix: files to port, checkpoint byte-format, commands). + +## M41 — Reusable deterministic CPU-parallel data generation *(2026-07-13; see `PARALLEL_SELFPLAY_PRD.md`)* — M41.1 + M41.2 + M41.3 ✅ SHIPPED + +**Why:** AlphaZero self-play (`SelfPlayCampaign`) generates games **single-threaded** (`TrainChunk`'s `for … PlayGame()`), +and for chess the wall-time is dominated by CPU-bound MCTS **movegen** — so a multi-core box mostly idles while training +crawls (the M40.4 256-sim run bottleneck). The GPU is a poor fit here (tiny net, batch-1 inference); **CPU-core +parallelism is the real lever.** + +**Finding (2-agent investigation):** the repo already parallelizes *cube* data-gen, but it's **hand-rolled + duplicated +in two Lab files** (`CubeImitationCampaign.cs:71`, `CubeEfficientCampaign.cs:93` — `Parallel.For` + per-worker lists + +per-worker seeded RNG); **nothing reusable exists in Core** (Core only has generic GEMM / `VectorEnv` / DAVI-featurize +parallelism). Self-play + DQN campaigns have no explicit parallelism. Self-play **is** safe to parallelize: shared +read-only net inference is concurrent-safe (fresh buffers, no static cache, `[ThreadStatic]` `NoGrad`, batch-1 forwards +don't nest-parallelize); the only blockers are the shared `_window`/counters and the shared mutable RNGs. The +bitwise-reproducibility invariant (M25/M26/**M36 SHA-verified**) is preservable exactly as Core already does it +(`VectorEnv`: per-unit RNG; GEMM: disjoint output rows → DOP-invariant). + +- **M41.1 ✅ — Core primitive.** `DeterministicParallel.Generate(count, seeds, stream, baseIndex, makeItem, parallel, dop)` + in `Core/Training`: per-index-derived RNG (golden-ratio stride, à la `VectorEnv`), disjoint ordered slots, index-order + results. 12 unit tests prove bitwise parallel==sequential across DOP 1/2/4/8/16 + edges (commit `bc0c48f`). +- **M41.2 ✅ — parallel self-play.** `SelfPlayCampaign` generation refactored onto it: each game a pure fn of its own + index-derived RNG over the stable read-only net, returning samples; owner-thread merge in ascending index then train. + Shuffle moved to the Buffer stream (no seed collision with game 0). `--parallel`/`--dop` on `ChessLab`. **Gate MET:** + a Connect-4 run gives a **byte-identical checkpoint** at sequential vs parallel-dop-1 vs dop-8 + (`SelfPlayCampaignTests.ParallelGeneration_...AtAnyDop`). Eval/arena left sequential (not the bottleneck; can't affect + weights). Commit `a9fa5c3`. +- **M41.3 ✅ — cube dedup.** `CubeImitationCampaign` + `CubeEfficientCampaign` migrated off their hand-rolled + `Parallel.For` onto `DeterministicParallel.Generate` (new raw-`ulong baseSeed` overload; the `SeedSequence` overload + now delegates to it). Passing `(baseSeed: roundBase, baseIndex: 1)` reproduces the old `roundBase + φ·(worker+1)` + per-worker seeding **byte-for-byte** — verified by `DeterministicParallelTests.RawSeedOverload_ReproducesTheCube…`, + so cube training output is unchanged. The shared `Interlocked` solve-counter is gone (each generator returns its own + count, summed on the owner thread). + +**Answers the owner's question:** the parallelism isn't in Core for no principled reason — it was hand-rolled per Lab +campaign; the pattern is generic and matches Core's existing determinism approach, so it should be (and now will be) +extracted. **Non-goals:** GPU/batched-MCTS (separate, larger effort); parallelizing the DQN campaigns (not the bottleneck). + +## M42 — Convolutional residual net for chess (reusable in Core) *(2026-07-13; see `RESIDUAL_CONV_NET_PRD.md`)* — M42.1 + M42.2 ✅ SHIPPED · M42.3 ⏳ training · M42.4 🔜 (gated on M42.3) + +**Why:** chess self-play has **plateaued at ~random** (M40.4: winRate-vs-random ~50%→35%, material margin flat ~+0.1 +of the +0.75 gate, no tier ever promotes) despite 256 sims + material-shaped targets. The honest bottleneck is the +**model**: a flat `[256,256]` MLP over a 1152-float vector throws away the 8×8 board structure. Owner's decision +(2026-07-13): a true **AlphaZero-style convolutional residual tower** over `[18,8,8]`. Stacked after M41 (which makes +the heavier training iterations affordable). + +**Finding (2-agent investigation):** a residual net *class* (`ResidualMlp`) is **already in Core** but is the wrong +shape (single scalar head, residual **MLP** not conv, used only by cube DAVI) — so "move it to the library" is a no-op; +it's already there. The reusable two-headed net (`PolicyValueNet`) is a flat MLP. **No convolution exists anywhere** +(no Conv2D/im2col/pool/BatchNorm; LayerNorm does exist and is the repo's deliberate choice). `SelfPlayCampaign`/ +`PolicyValueTraining` hardcode the concrete `PolicyValueNet` type — there's **no two-headed-net interface** — so a conv +net needs one introduced. The obs already reshapes cleanly to `[18,8,8]` (`plane*64+sq`). And the **browser-inference +twin** (`chess_solver.pg` `PgPolicyValueNet`) is a flat-MLP forward — a conv net breaks client-side play until the `.pg` +gains a conv forward (inference-only), so that's a first-class phase, not a follow-up. + +- **M42.1 ✅ — Conv2D in Core.** `Tensor.Conv2D` via **im2col → existing GEMM → col2im** (reuses the tuned, GPU-routed + GEMM; NO new backend/ILGPU kernel), rank-2 `[N, C·H·W]` representation so `CheckRank2` is never tripped, LayerNorm (not + BatchNorm). **Gate MET:** three finite-difference gradient checks (3×3 SAME, stride-2 valid, 1×1) green + (`GradCheckTests.Conv2D_*`). Commit `67806af`. +- **M42.2 ✅ — `IPolicyValueNet` + the conv net.** Interface introduced; `PolicyValueNet` implements it (self-play + determinism gate still byte-identical ⇒ zero MLP behaviour change). `ConvResidualPolicyValueNet` (3×3 stem → N residual + blocks → 1×1-conv policy + value heads; Save/Load/LayerActivations). `SelfPlayCampaign`/`PolicyValueTraining` + generalized to the interface + an `IPolicyValueNetBuilder` (MlpNetBuilder default / ConvNetBuilder); `ChessLab --arch + conv --filters --blocks`. **Gate MET** (`ConvResidualNetTests`): head shapes, exact save/load round-trip, loss falls + under Adam. (De-risking residual-MLP step skipped — went straight to conv.) Commit `21b779d`. +- **M42.3 ⏳ — train chess with the conv net.** `--arch conv --filters 64 --blocks 6` + material shaping + ladder, + running in the background. **Gate:** beats the MLP baseline — material margin ≥ +0.75 and/or winRate ≫ 50% with **≥1 + ladder tier promoted**; determinism preserved. *(Long run; evaluated from the background training, not blocking.)* +- **M42.4 🔜 (gated on M42.3) — browser conv forward + parity.** Add an inference-only conv2d forward to + `chess_solver.pg` + teach `chess-net.ts` the conv `.ckpt` layout; regenerate the C#/TS twin. **Gate:** + `ChessNetParityTests` green on conv `.ckpt`; `/chess` plays a shipped conv tier. **Deliberately not started until M42.3 + proves the conv net is worth shipping** (PRD risk #2 — don't port to the browser a net that doesn't beat the baseline). + +**Non-goals:** removing `PolicyValueNet` (stays the connect-4/cube-policy/rush-hour net + fast baseline) or `ResidualMlp` +(stays the cube DAVI value net); spatial BatchNorm (reuse LayerNorm); Net2Net growth for the conv net (stays MLP-only). + ## Testing strategy (cross-cutting, from research) 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket. diff --git a/docs/prd/RESIDUAL_CONV_NET_PRD.md b/docs/prd/RESIDUAL_CONV_NET_PRD.md new file mode 100644 index 0000000..3c7a5eb --- /dev/null +++ b/docs/prd/RESIDUAL_CONV_NET_PRD.md @@ -0,0 +1,109 @@ +# Convolutional residual policy/value net (Core) + chess adoption — PRD + +**Status:** M42.1 + M42.2 ✅ SHIPPED 2026-07-13 (branch `m39-chess-selfplay-plan`, PR #32) · M42.3 ⏳ training · M42.4 🔜 gated on M42.3. +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M42 · **Depends on:** M41 (parallel self-play — makes the training iterations this needs affordable) · **Supersedes** the "flat MLP is the ceiling" note in [CHESS_WEB_POLYGLOT_PRD.md](CHESS_WEB_POLYGLOT_PRD.md) and [CHESS_SELFPLAY_PRD.md](CHESS_SELFPLAY_PRD.md). + +## Implementation status (what actually shipped vs. this design) +- **M42.1** `Tensor.Conv2D` (commit `67806af`) — im2col → existing GEMM → col2im, in `Core/Numerics/TensorConv.cs`. + Confirmed **no `IComputeBackend`/ILGPU change was needed** (the design's preferred route). Rank-2 `[N, C·H·W]`, LayerNorm + reused. Gate MET: 3 finite-difference gradient checks (3×3 SAME, stride-2 valid, 1×1). +- **M42.2** `IPolicyValueNet` + `ConvResidualPolicyValueNet` + arch-agnostic campaign (commit `21b779d`). Per §4b/§4c, + with an `IPolicyValueNetBuilder` (Mlp/Conv) as the arch factory. **The de-risking two-headed-residual-MLP checkpoint + (§M42.2) was SKIPPED** — went straight to the conv net. Gate MET: head shapes, exact save/load round-trip, loss falls; + MLP self-play determinism gate still byte-identical (zero behaviour change). +- **M42.3** ⏳ conv training (`--arch conv --filters 64 --blocks 6`) — runs offline via the Lab; owner-driven (paused + during PR merge, to be restarted). Gate unchanged: beat the MLP baseline / ≥1 ladder tier promotes. +- **M42.4** 🔜 browser conv forward + parity — **not started, deliberately gated on M42.3** proving the conv net beats + the baseline (PRD risk #2). The current shipped browser demo uses the flat-MLP net (committed at `chess.az.d1.ckpt`). + +## 1. Problem + +Chess self-play has **plateaued at ~random play** despite 256-sim MCTS and material-shaped value targets (M40.4). The live run stalls at winRate-vs-random ~50% (drifting to 35%), material margin ~+0.1 pawns (gate needs +0.75), and policy loss barely moving (2.35→2.22 over 500 games) — **no tier ever promotes.** The honest diagnosis (recorded across the chess PRDs and the training memory): the model is the bottleneck, not search depth or reward shaping. The net is a **flat `[256,256]` ReLU MLP over a 1152-float vector** (`PolicyValueNet`) — it throws away the board's 8×8 spatial structure, so it cannot learn the local piece-interaction patterns (pins, forks, pawn chains, king safety) that chess play is built on. + +**The owner's decision (2026-07-13): give chess a true AlphaZero-style convolutional residual net** — reshape the observation to `[18, 8, 8]` and run a spatial residual tower. This PRD covers building that net *reusably in Core* (so any board game can use it) and adopting it for chess, including the browser-inference twin. + +## 2. Findings (from a 2-agent read-only investigation, 2026-07-13) + +### 2a. What the chess net is today +`PolicyValueNet` (`src/…Core/Nn/PolicyValueNet.cs:15`) — a shared variable-depth ReLU **MLP** trunk (`Linear[]`) + two **linear** heads (policy logits `[B,4672]`, scalar value `[B,1]`); `tanh` on the value is applied by the *trainer* (`PolicyValueTraining.cs:24`), not the net. Chess builds it with trunk `[256,256]` (`ChessLab.cs:20`, `SelfPlayCampaign.cs:78/109`). It is the same net `CubePolicyNet`/`RushHourPolicyNet` wrap — all flat MLPs. + +### 2b. Is a residual/conv net already reusable? Partly — and not usable as-is. +- **A residual net class IS already in Core:** `ResidualMlp` (`src/…Core/Nn/ResidualMlp.cs:21`) — pre-activation residual blocks `x → x + W₂·ReLU(LayerNorm(W₁·x))`. **But** it's single-headed **scalar** output (`IValueNet`, cost-to-go), residual **MLP** (not conv), and used only by the cube DAVI Lab campaign (`CubeDaviCampaign.cs:141`). It **cannot** serve as a chess policy/value net (no policy head, no spatial structure). So "move the residual net to the library" is a **no-op — it's already there**; the missing thing is a *two-headed convolutional* net, which does not exist. +- **No convolution exists anywhere** in Core, Environments, Lab, or the Polyglot `.pg` files. **No** Conv2D, **no** im2col/col2im, **no** pooling, **no** BatchNorm. **LayerNorm does exist** (`TensorOps.cs:272`, autograd op with learnable γ/β) and the codebase deliberately prefers it over BatchNorm (`ResidualMlp.cs:6-13`). +- **The differentiable op set** (`TensorOps.cs`): `MatMul`, `Add`, `AddBias`, `Sub`, `Mul`, `MulScalar`, `Relu/Tanh/Exp/Log/Square`, `Clamp`, `Min`, `Gather`, `Sum`, `Mean`, `SumRows`, `LogSoftmax`, `HuberLoss`, `LayerNorm`, `MseLoss`, **`Reshape`** (buffer-sharing, grad pass-through). Skip-add = plain `Add`. The backend seam (`IComputeBackend`, managed + ILGPU) exposes 3 GEMM layouts, elementwise Map/Zip, reductions, LayerNorm — **but no conv/pool/batchnorm kernels.** + +### 2c. The structural blocker +`SelfPlayCampaign` and `PolicyValueTraining` reference the **concrete `PolicyValueNet` type**, not an interface (`SelfPlayCampaign.cs:45,57,102,109,282,405,436,445`; `PolicyValueTraining.cs:18`). There is **no abstraction for "a two-headed policy/value net."** A different net class can't be dropped in without introducing that interface and generalizing the campaign/trainer. This is the main refactor cost — small, but it gates everything. + +### 2d. The observation is already conv-ready +Chess obs = **18 planes × 64 = 1152 floats**, laid out `plane*64 + sq` with `sq = rank*8 + file` (`chess_solver.pg:405-427`; C# mirror `ChessGame.cs:53`): planes 0–5 white P/N/B/R/Q/K, 6–11 black, 12 side-to-move, 13–16 castling, 17 en-passant. → **trivially reshapes to `[C=18, H=8, W=8]`.** Policy head target = 4672 (64×73 AlphaZero move encoding), value = 1 scalar. So the conv net's contract is unchanged: `(Logits[B,4672], Value[B,1])`. + +## 3. Decision + +**Build a reusable two-headed convolutional residual net in Core, gated behind a new `IPolicyValueNet` interface, and adopt it for chess** — including a conv forward in the browser-inference `.pg` twin. Keep the existing flat `PolicyValueNet` as a selectable architecture (it's the connect-4 / cube-policy / rush-hour net and the fast baseline). Do **not** remove `ResidualMlp` — it stays the cube DAVI value net. + +## 4. Design + +### 4a. New Core ops — Conv2D (the only genuinely new numerics) +Implement convolution as **im2col → existing GEMM → col2im**, so the heavy math reuses the already-tuned (and GPU-routed) GEMM instead of a bespoke kernel: +- **`IComputeBackend` (managed + ILGPU) + `TensorOps`:** add a differentiable `Conv2D(input, weight, bias, inC, H, W, outC, kernel, stride, pad)`. + - **Representation:** keep tensors **rank-2** — input `[N, inC·H·W]`, output `[N, outC·outH·outW]` — with the conv op carrying `(inC,H,W,…)` explicitly. This **sidesteps the `CheckRank2` asserts entirely** (no 4-D tensor ever reaches `MatMul`/`LayerNorm`); `Reshape` is only used to re-view between conv and the flat heads. + - **Forward:** `im2col(x)` → `[N·outH·outW, inC·k·k]`, GEMM with `weight [inC·k·k, outC]`, add bias → output. im2col/col2im are cheap index gathers (managed loops); the GEMM routes to the GPU via the adaptive backend for free. + - **Backward:** dInput = `col2im(dOut · Wᵀ)`; dWeight = `im2col(x)ᵀ · dOut`; dBias = column-sum of dOut. All three reuse the existing GEMM layouts + `AddInto`/`BiasGradInto`. +- **Normalization:** reuse the existing **LayerNorm** (proven stable in `ResidualMlp`) rather than building spatial BatchNorm — one fewer new kernel, and consistent with the repo's deliberate LayerNorm choice. (Spatial BatchNorm is a possible later refinement, out of scope.) +- **Pooling:** not needed — AlphaZero towers are all-conv with a flatten+Linear at each head. +- **Gate:** finite-difference gradient check on `Conv2D` (dInput, dWeight, dBias) and a forward-value check vs a hand-computed small example; determinism (same output at any backend DOP, like GEMM). + +### 4b. New Core net — `ConvResidualPolicyValueNet` (`Core/Nn/`) +AlphaZero-standard, parameterized by `(inputPlanes, boardH, boardW, filters, blocks, actions)`: +- **Stem:** `Conv2D 3×3 (inC→filters, pad 1)` → LayerNorm → ReLU. +- **Tower:** `blocks` × residual block `[Conv3×3 → LN → ReLU → Conv3×3 → LN → (+skip) → ReLU]` (all `filters→filters`, pad 1, so the spatial shape is preserved and the skip-add is same-shape). +- **Policy head:** `Conv2D 1×1 (filters→2)` → LN → ReLU → flatten → `Linear(2·H·W → actions)`. +- **Value head:** `Conv2D 1×1 (filters→1)` → LN → ReLU → flatten → `Linear(H·W → filters)` → ReLU → `Linear(filters → 1)` (linear; `tanh` stays in the trainer, matching `PolicyValueNet`). +- Implements **`IPolicyValueNet`** (§4c); `Save`/`Load` with a new checkpoint kind (e.g. `selfplay-pv-conv`), storing `(filters, blocks)` + all layer floats, matching `CheckpointFormat`. Provides `LayerActivations` for the `--viz` telemetry seam. + +### 4c. The abstraction — `IPolicyValueNet` (the enabling refactor) +Introduce in Core: +```csharp +public interface IPolicyValueNet +{ + (Tensor Logits, Tensor Value) Forward(Tensor observations); + IEnumerable Parameters(); + float[][] LayerActivations(Tensor observation); + void Save(Stream destination, string kind); +} +``` +- `PolicyValueNet` implements it verbatim (it already has every member — **zero behaviour change**, SHA-identical checkpoints). +- `SelfPlayCampaign` + `PolicyValueTraining` are generalized from the concrete type to `IPolicyValueNet` (field type + the evaluator/arena forwards). Net **construction** and **loading** go through a small architecture factory keyed on an `--arch` value (so the right `Load` runs for a given checkpoint kind). +- **Net2Net growth** (`WidenTo`/`Deepen`) stays **MLP-specific** and out of the interface — chess self-play doesn't grow the net (fixed `--hidden`); if a conv-growth path is ever wanted it's a separate optional interface. (Verify no growth call sits on the campaign path during M42.2; the investigation found none.) + +### 4d. Chess adoption (Lab) +- `ChessLab`: `--arch mlp|conv` (default stays `mlp` until the conv net proves out), plus `--filters` (e.g. 64) and `--blocks` (e.g. 6). `SelfPlayCampaign` builds the chosen `IPolicyValueNet`. +- All of M40.4 (material-shaped value target, the auto-capture ladder, `--viz`) is architecture-agnostic and works unchanged. + +### 4e. Browser-inference twin — conv forward in `chess_solver.pg` (do not skip) +The whole M40 browser story is single-source Polyglot inference: `PgPolicyValueNet.forward` in `chess_solver.pg` is a **flat-MLP** forward, and `chess-net.ts` parses the flat `.ckpt`. A conv net **breaks client-side play** until the `.pg` gains a **conv2d forward** (inference only — no backward, so it's straightforward: nested bounded `for` over out-channels × spatial × kernel, using `Math` from `std.math`) and the `.ckpt` parser learns the conv layout. This is transpiled to both C# and TS from the one `.pg`, and re-gated by the existing `ChessNetParityTests` (C# `Forward` vs generated net on real `.ckpt` bytes). **This is a first-class phase (M42.4), not a follow-up** — shipping stronger weights the browser can't run would regress M40. + +## 5. Phases + +- **M42.1 — Conv2D in Core.** `Conv2D` op (im2col+GEMM+col2im) in `IComputeBackend` (managed + ILGPU) + `TensorOps`, rank-2 representation. **Gate:** finite-difference gradient check (dInput/dWeight/dBias) + forward value check + DOP-invariant output. +- **M42.2 — `IPolicyValueNet` + the conv net, in Core.** Introduce the interface; `PolicyValueNet` implements it (SHA-identical checkpoints — proves no behaviour change). Build `ConvResidualPolicyValueNet` (Save/Load/LayerActivations). Generalize `SelfPlayCampaign`/`PolicyValueTraining` + an arch factory. **Gate:** existing self-play tests green with MLP unchanged; a train-one-batch smoke test on the conv net (loss decreases); connect-4 still trains identically. + - **De-risking checkpoint (recommended, cheap):** before the conv tower, wire a *two-headed residual **MLP*** (add a policy head to the `ResidualMlp` pattern — **no new kernels**) behind the same interface and run a short chess train. If depth+residuals alone don't move the plateau, that's a fast signal about targets/search *before* investing in conv. Not a substitute for the conv net — a smoke test that de-risks it. +- **M42.3 — train chess with the conv net.** `--arch conv --filters 64 --blocks 6` (tune), material shaping + ladder on. **Gate:** clearly beats the flat-MLP baseline — material margin ≥ +0.75 pawns and/or winRate-vs-random well above 50% and **at least one ladder tier promotes**; report the training curve vs the MLP plateau. Reproducibility (SHA dop-1==dop-N from M41) preserved. +- **M42.4 — browser conv forward + parity.** Add a conv2d forward to `chess_solver.pg` (inference-only) + teach `chess-net.ts` the conv `.ckpt` layout; regenerate the C#/TS twin. **Gate:** `ChessNetParityTests` green (C# vs generated within f32 tol on real conv `.ckpt` bytes); the `/chess` page plays with a shipped conv tier. + +## 6. Risks +1. **Conv correctness** — mitigated by the finite-difference gradient gate (M42.1) before any net is built on it. +2. **Conv net still doesn't beat the plateau** — possible if the block is targets/search, not capacity; the M42.2 residual-MLP de-risking checkpoint surfaces this cheaply. Fallback levers if so: more sims, longer training (M41 makes this affordable), or revisiting the value target. +3. **Training cost** — a conv tower is far heavier than the MLP; **this is exactly why M41 (parallel self-play) lands first.** The im2col+GEMM route keeps the heavy math on the GPU-routed GEMM. +4. **Browser perf** — a conv forward per MCTS node in JS may be slow; if so, cap browser sims per difficulty tier (the tier system already sets per-tier sims) or ship a smaller `filters/blocks` for the web than for the strongest offline tier. Flag during M42.4. +5. **Checkpoint/interface churn** — the `IPolicyValueNet` refactor touches the shared self-play path; the SHA-identical-MLP gate (M42.2) guards against a silent regression to connect-4/chess. + +## 7. Verification +- M42.1: conv gradient (finite-diff) + forward-value unit tests; DOP-invariance. +- M42.2: full suite green with MLP behaviour SHA-identical; conv train-one-batch loss decrease; connect-4 unchanged. +- M42.3: chess conv run **beats the MLP baseline** on material margin / winRate with ≥1 tier promoted; determinism preserved. +- M42.4: `ChessNetParityTests` green on conv `.ckpt`; `/chess` plays a conv tier end-to-end. + +See [PLAN.md](PLAN.md) M42. diff --git a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md index 0ac6468..28df398 100644 --- a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md +++ b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md @@ -1,11 +1,21 @@ # Polyglot bug handoff — incremental re-transpile emits a duplicate prelude / non-`partial` PolyglotProgram +> **✅ RESOLVED 2026-07-13 in `MintPlayer.Polyglot.MSBuild` 0.6.0 (PR #26).** The verified stamp-`Outputs` + +> `RemoveDir` fix (see the 2026-07-12 update below) shipped upstream. This repo is on 0.6.0; the local +> `_PolyglotForceFullRetranspile` workaround and the `MintPlayer.Polyglot.MSBuild.targets.FIXED` drop-in were +> removed. Kept for history / root-cause reference. + **Found:** 2026-07-10, during Snake M34 (adding look-ahead search to `snake_solver.pg`). -**Polyglot:** `MintPlayer.Polyglot.MSBuild` **0.3.1** (transpiler `tools/win-x64/polyglot.exe` 0.3.1). -**Polyglot source:** `C:\Repos\MintPlayer.Polyglot` +**Polyglot:** `MintPlayer.Polyglot.MSBuild` **0.3.1** — **still present in 0.5.3** (verified 2026-07-12; see the update at +the bottom). This is an **MSBuild `.targets` bug, not a CLI/version bug**, so bumping the package does not fix it. +**Polyglot source:** `C:\Repos\MintPlayer.Polyglot` (fix lives in `build/MintPlayer.Polyglot.MSBuild.targets`). **Severity:** breaks *incremental* dev builds after editing a `.pg` in a **multi-`.pg` project**; a **clean build always succeeds**, so CI / fresh clones are unaffected — but it's confusing and easy to misdiagnose. +> **⚠️ 2026-07-12 update at the bottom of this file** sharpens the root cause (MSBuild *partial-incremental build* +> passes a **subset** of `.pg` to the CLI) and gives a **verified** `.targets` fix. Note the original "clean the `--out` +> dir" idea in *Fix ideas* below is **insufficient and can be harmful** on its own — read the update first. + ## Symptom In a project with **2+ `.pg` files** built by the glob-all MSBuild target (here: `fruitcake_solver.pg`, @@ -61,3 +71,101 @@ Note: the M34 search *does* use a top-level `record PgSnakeBeamNode` (a typed fr separate TS `any`-inference failure; see `SNAKE_SEARCH_PRD.md` §9). Records build fine on a clean build (FruitCake uses them too); this incremental-staleness bug is unrelated to records. If you hit the errors above during iterative `.pg` editing: `rm src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs` then rebuild. + +--- + +## UPDATE 2026-07-12 — precise root cause + a verified fix (bug persists through 0.5.3) + +Re-encountered during **Chess M40.1** (adding a 4th solver, `chess_solver.pg`). Bumping the package **0.3.1 → 0.5.3** +did **not** fix it — an incremental touch of one `.pg` still produces `CS0101`/`CS0260`/`CS8863`. That rules out the +CLI: **the bug is in the MSBuild `.targets`, and it is version-independent.** + +### The actual trigger (sharper than "stale files in the out dir") + +`PolyglotTranspile` declares a **per-file** output map: + +```xml +Inputs="@(PolyglotFile);$(PolyglotTool)" +Outputs="@(PolyglotFile->'$(PolyglotOutDir)%(Filename).cs')" +``` + +Because both `Inputs` and `Outputs` are item transforms of the same `@(PolyglotFile)`, MSBuild does a **partial +incremental build**: when only *one* `.pg` is newer than its `.cs`, MSBuild runs the target with `@(PolyglotFile)` +**filtered to just that one file**. The `Exec` then invokes the CLI with a **single** source: + +``` +polyglot build "…/chess_solver.pg" --target csharp --out "…/polyglot/." +``` + +From the CLI's point of view that is a **single-module** build, so it (correctly, for a lone module) emits +`chess_solver.cs` in **standalone** mode — inline `Option`/`Some`/`None` + a **non-`partial`** `PolyglotProgram`. That +standalone unit then collides with the **other** solvers' `.cs` + the shared `__polyglot_prelude.cs` still sitting in +`obj/` from the prior full build → `CS0260`/`CS0101`/`CS8863`. The stale prelude is a *symptom*, not the cause; the +cause is **MSBuild handing the CLI a subset**. (`rm *.cs` "fixes" it only because it makes *all* outputs missing, which +forces MSBuild to re-run with the **full** set.) + +### Why the earlier "clean the `--out` dir" idea is not enough — and is harmful alone + +If the CLI cleaned `--out` at the start of every build but MSBuild still invoked it with a **subset**, then a +single-file incremental build would **delete the other solvers' generated `.cs`** and emit only the changed one → +"type `PgFruitCakeWorld` not found" etc. The out-dir cleaning must be paired with **always invoking the CLI with the +full set**. The real fix is therefore MSBuild-side. + +### Verified fix (MSBuild `.targets`) — force all-or-nothing with a single stamp output + +Replace `PolyglotTranspile`'s per-file `Outputs` with a **single stamp file** (a static path, *not* an item transform), +and clean the out dir so a removed/renamed `.pg` leaves no orphan `.cs`. A single static `Outputs` makes the target +**un-batchable**, so any staleness re-runs it with the **complete** `@(PolyglotFile)`: + +```xml + + … (unchanged _PolyglotLibArg / _PolyglotAccessArg) … + + + + + +``` + +`_PolyglotAddGenerated` already globs `$(PolyglotOutDir)*.cs`, so the `.stamp` is not compiled; add it to `FileWrites` +too so `dotnet clean` removes it. A no-op rebuild stays up-to-date (stamp newer than every `.pg` and the tool); any +`.pg`/tool change re-runs the full set. A defensive `--clean-out`/idempotent-prelude on the CLI is a fine +belt-and-braces addition but is not sufficient on its own (see above). + +**This exact patch is VERIFIED (2026-07-12), not just proposed.** A drop-in copy of the fixed target file is staged +alongside this doc: **`docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED`** — its only diff vs the +current upstream `build/MintPlayer.Polyglot.MSBuild.targets` is the four lines above (stamp `Outputs`, `RemoveDir`, +`Touch`, `FileWrites` stamp) plus the rationale comment. Session B in `C:\Repos\MintPlayer.Polyglot` can copy it over +`src/MintPlayer.Polyglot.MSBuild/build/MintPlayer.Polyglot.MSBuild.targets`, then branch + PR. + +**How it was verified — a 2-`.pg` repro harness importing the source `.targets` directly** (no NuGet cache mutation): + +```bash +D=/tmp/pgfix; mkdir -p "$D"; cd "$D" +printf 'fn aHello(): i32 => 1\n' > a.pg +printf 'fn bHello(): i32 => 2\n' > b.pg +# Test.csproj: net10.0 project that s the .props then the .targets under test, and sets +# to ~/.nuget/packages/mintplayer.polyglot.msbuild/0.5.3/tools/win-x64/polyglot.exe +dotnet build -c Release # clean build: succeeds +touch a.pg && dotnet build -c Release # UNPATCHED .targets → a.cs emitted standalone → CS0101/CS0260/CS8863 +``` + +Against the **unpatched** source `.targets` the `touch a.pg` rebuild fails exactly as reported (`a.cs:7` non-`partial` +PolyglotProgram + duplicate prelude). Against the **patched** `.targets` (the `.FIXED` file): the same `touch` +rebuild **succeeds**, a no-op rebuild logs *"Skipping target PolyglotTranspile because all output files are +up-to-date"* (incrementality preserved), a touch logs *"Touching …/__polyglot.stamp"* (full re-transpile), and a fully +clean build succeeds. All four states green. + +### Consumer-side mitigation already shipped (so this repo no longer needs the `rm`) + +Until the `.targets` ships the fix, `MintPlayer.AI.ReinforcementLearning.Environments.csproj` carries a +`_PolyglotForceFullRetranspile` target that reproduces the effect without editing the package: it wipes +`$(IntermediateOutputPath)polyglot\` whenever any `.pg` is newer than the shared `__polyglot_prelude.cs`, forcing +`PolyglotTranspile` to regenerate the whole set in one CLI call. Verified: the incremental touch that reliably broke now +rebuilds clean in Release **and** Debug; no-op rebuilds stay up-to-date; full test suite 362/362, chess perft 25/25. +When the upstream `.targets` fix lands, that local target can be deleted. diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ConvResidualPolicyValueNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ConvResidualPolicyValueNet.cs new file mode 100644 index 0000000..5f9675a --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ConvResidualPolicyValueNet.cs @@ -0,0 +1,193 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Nn; + +/// +/// An AlphaZero-style two-headed policy/value network over a spatial board (PLAN M42): the observation is a stack of +/// planes feature maps of size boardH×boardW, carried as a rank-2 [B, planes·H·W] tensor. A 3×3 conv +/// stem lifts it to filters channels, a tower of blocks residual blocks +/// (x → relu(x + LN(conv(relu(LN(conv(x))))))) processes it at constant H×W, then a 1×1-conv policy head → a +/// linear to the flat action space, and a 1×1-conv value head → two linears to a scalar. Every spatial layer keeps +/// H×W (stride 1, pad 1 for 3×3, pad 0 for 1×1), so the residual skip is always shape-matched. +/// +/// Unlike the flat , this preserves the board's 2-D structure, so it can learn local +/// piece-interaction patterns (the lever the flat MLP lacked — see PLAN M40.4 plateau). Normalization is +/// over each sample's whole feature map (not BatchNorm), matching the repo's +/// deliberate choice for stability under a moving target (). The value head stays linear; +/// the trainer applies tanh. Implements , so it drops into the self-play stack in +/// place of the flat net. +/// +/// +public sealed class ConvResidualPolicyValueNet : IPolicyValueNet +{ + private readonly int _planes, _h, _w, _actions, _filters, _blocks; + + private readonly Conv _stem; + private readonly Norm _stemNorm; + private readonly Conv[] _conv1, _conv2; // the two convs of each residual block + private readonly Norm[] _norm1, _norm2; + private readonly Conv _policyConv; // 1×1 filters→2 + private readonly Norm _policyNorm; + private readonly Linear _policyHead; // 2·H·W → actions + private readonly Conv _valueConv; // 1×1 filters→1 + private readonly Norm _valueNorm; + private readonly Linear _valueHidden, _valueHead; // H·W → filters → 1 + + private const int PolicyHeadChannels = 2; + + public ConvResidualPolicyValueNet(int planes, int boardH, int boardW, int actions, int filters, int blocks, + Xoshiro256StarStar rng) + { + if (blocks < 1) throw new ArgumentException("A residual tower needs at least one block."); + if (filters < 1) throw new ArgumentException("A conv net needs at least one filter."); + _planes = planes; _h = boardH; _w = boardW; _actions = actions; _filters = filters; _blocks = blocks; + int hw = boardH * boardW; + + _stem = new Conv(planes, filters, k: 3, pad: 1, rng); + _stemNorm = new Norm(filters * hw); + + _conv1 = new Conv[blocks]; _conv2 = new Conv[blocks]; + _norm1 = new Norm[blocks]; _norm2 = new Norm[blocks]; + for (int i = 0; i < blocks; i++) + { + _conv1[i] = new Conv(filters, filters, k: 3, pad: 1, rng); + _norm1[i] = new Norm(filters * hw); + _conv2[i] = new Conv(filters, filters, k: 3, pad: 1, rng); + _norm2[i] = new Norm(filters * hw); + } + + _policyConv = new Conv(filters, PolicyHeadChannels, k: 1, pad: 0, rng); + _policyNorm = new Norm(PolicyHeadChannels * hw); + _policyHead = new Linear(PolicyHeadChannels * hw, actions, rng, Activation.None); + + _valueConv = new Conv(filters, 1, k: 1, pad: 0, rng); + _valueNorm = new Norm(hw); + _valueHidden = new Linear(hw, filters, rng, Activation.Relu); + _valueHead = new Linear(filters, 1, rng, Activation.None); + } + + public int InputSize => _planes * _h * _w; + public int Actions => _actions; + public string Describe() => $"conv {_filters}f×{_blocks}b ({_planes}×{_h}×{_w})"; + + public (Tensor Logits, Tensor Value) Forward(Tensor observations) + { + var x = _stem.Forward(observations, _h, _w).LayerNorm(_stemNorm.Gamma, _stemNorm.Beta).Relu(); + for (int i = 0; i < _blocks; i++) + { + var h = _conv1[i].Forward(x, _h, _w).LayerNorm(_norm1[i].Gamma, _norm1[i].Beta).Relu(); + h = _conv2[i].Forward(h, _h, _w).LayerNorm(_norm2[i].Gamma, _norm2[i].Beta); + x = x.Add(h).Relu(); + } + + var p = _policyConv.Forward(x, _h, _w).LayerNorm(_policyNorm.Gamma, _policyNorm.Beta).Relu(); + var logits = _policyHead.Forward(p); + + var v = _valueConv.Forward(x, _h, _w).LayerNorm(_valueNorm.Gamma, _valueNorm.Beta).Relu(); + var value = _valueHead.Forward(_valueHidden.Forward(v).Relu()); + return (logits, value); + } + + public float[][] LayerActivations(Tensor observation) + { + var acts = new List(2 + _blocks); + var x = _stem.Forward(observation, _h, _w).LayerNorm(_stemNorm.Gamma, _stemNorm.Beta).Relu(); + acts.Add([.. x.Data]); + for (int i = 0; i < _blocks; i++) + { + var h = _conv1[i].Forward(x, _h, _w).LayerNorm(_norm1[i].Gamma, _norm1[i].Beta).Relu(); + h = _conv2[i].Forward(h, _h, _w).LayerNorm(_norm2[i].Gamma, _norm2[i].Beta); + x = x.Add(h).Relu(); + acts.Add([.. x.Data]); + } + var (logits, value) = Forward(observation); + acts.Add([.. logits.Data]); + acts.Add([.. value.Data]); + return [.. acts]; + } + + // Enumerated in a fixed order so the optimizer and the checkpoint round-trip agree. + public IEnumerable Parameters() + { + foreach (var p in _stem.Parameters()) yield return p; + yield return _stemNorm.Gamma; yield return _stemNorm.Beta; + for (int i = 0; i < _blocks; i++) + { + foreach (var p in _conv1[i].Parameters()) yield return p; + yield return _norm1[i].Gamma; yield return _norm1[i].Beta; + foreach (var p in _conv2[i].Parameters()) yield return p; + yield return _norm2[i].Gamma; yield return _norm2[i].Beta; + } + foreach (var p in _policyConv.Parameters()) yield return p; + yield return _policyNorm.Gamma; yield return _policyNorm.Beta; + foreach (var p in _policyHead.Parameters()) yield return p; + foreach (var p in _valueConv.Parameters()) yield return p; + yield return _valueNorm.Gamma; yield return _valueNorm.Beta; + foreach (var p in _valueHidden.Parameters()) yield return p; + foreach (var p in _valueHead.Parameters()) yield return p; + } + + // ── Checkpoint ──────────────────────────────────────────────────────────────────────────────────────────────── + // v1: [planes, boardH, boardW, filters, blocks] then every parameter's floats in Parameters() order. + private const int Version = 1; + + public void Save(Stream destination, string kind) + { + using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.WriteHeader(writer, kind, Version); + CheckpointFormat.WriteInts(writer, [_planes, _h, _w, _filters, _blocks]); + foreach (var p in Parameters()) CheckpointFormat.WriteFloats(writer, p.Data); + } + + /// Loads a conv policy/value net. comes from the environment (never stored); + /// the spatial + tower shape comes from the file, so a shipped checkpoint reconstructs exactly. + public static ConvResidualPolicyValueNet Load(Stream source, string kind, int actions) + { + using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.ReadHeader(reader, kind, Version); + int[] dims = CheckpointFormat.ReadInts(reader); // planes, H, W, filters, blocks + var net = new ConvResidualPolicyValueNet(dims[0], dims[1], dims[2], actions, dims[3], dims[4], + new Xoshiro256StarStar(0)); + foreach (var p in net.Parameters()) + CheckpointFormat.ReadFloats(reader).CopyTo(p.Data.AsSpan()); + return net; + } + + // ── Building blocks ─────────────────────────────────────────────────────────────────────────────────────────── + // A single conv layer (weight [inC·k·k, outC] + bias [outC]) that runs through the Conv2D autograd op. Spatial + // size is preserved (pad = (k-1)/2 for odd k, stride 1), so shapes chain and the residual skip stays aligned. + private sealed class Conv + { + private readonly Tensor _weight, _bias; + private readonly int _inC, _outC, _k, _pad; + + public Conv(int inC, int outC, int k, int pad, Xoshiro256StarStar rng) + { + float std = MathF.Sqrt(2f / (inC * k * k)); // He init (ReLU) + _weight = new Tensor(Tensor.RandomNormal(rng, 0f, std, inC * k * k, outC).Data, inC * k * k, outC) { RequiresGrad = true }; + _bias = new Tensor(new float[outC], outC) { RequiresGrad = true }; + _inC = inC; _outC = outC; _k = k; _pad = pad; + } + + public Tensor Forward(Tensor x, int h, int w) => x.Conv2D(_weight, _bias, _inC, h, w, _outC, _k, _k, 1, _pad); + + public IEnumerable Parameters() { yield return _weight; yield return _bias; } + } + + // LayerNorm scale/shift over a whole flattened feature map (γ=1, β=0), both learnable [size]. + private sealed class Norm + { + public Tensor Gamma { get; } + public Tensor Beta { get; } + + public Norm(int size) + { + Gamma = new Tensor(new float[size], size) { RequiresGrad = true }; + Array.Fill(Gamma.Data, 1f); + Beta = new Tensor(new float[size], size) { RequiresGrad = true }; + } + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/IPolicyValueNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/IPolicyValueNet.cs new file mode 100644 index 0000000..f1e09ad --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/IPolicyValueNet.cs @@ -0,0 +1,29 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Nn; + +/// +/// A two-headed policy/value network: an observation batch → raw policy logits [B, actions] + a scalar value [B, 1] +/// (the value is left linear; the trainer bounds it via tanh). This is the seam the AlphaZero self-play stack +/// (SelfPlayCampaign / PolicyValueTraining / the MCTS leaf evaluator) depends on, so alternative +/// architectures — the flat and the convolutional +/// — are interchangeable without the campaign knowing which it holds. Construction and checkpoint reload stay +/// architecture-specific (a net factory owns those); this interface is only what a trained net must offer at runtime. +/// +public interface IPolicyValueNet +{ + /// Batched forward pass (autograd-recorded): raw policy logits [B, actions] + value [B, 1]. + (Tensor Logits, Tensor Value) Forward(Tensor observations); + + /// Every trainable tensor, in a stable order (drives the optimizer and checkpoint round-trip). + IEnumerable Parameters(); + + /// Per-layer activations for one input row (the live-network viewer seam). + float[][] LayerActivations(Tensor observation); + + /// Serializes the net (the tag is supplied by the owner/factory). + void Save(Stream destination, string kind); + + /// A short human-readable shape summary for logs (e.g. "trunk [256,256]" or "conv 64f×6b"). + string Describe(); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs index b1e0220..da36da1 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs @@ -12,7 +12,7 @@ namespace MintPlayer.AI.ReinforcementLearning.Core.Nn; /// kind. The trunk is a [] (not a fixed pair) so the net can grow deeper as well as /// wider mid-training via the function-preserving transforms. /// -public sealed class PolicyValueNet +public sealed class PolicyValueNet : IPolicyValueNet { private readonly Linear[] _trunk; private readonly Linear _policyHead, _valueHead; @@ -40,6 +40,8 @@ public PolicyValueNet(int inputSize, int[] hidden, int actions, Xoshiro256StarSt /// The shared-trunk hidden widths (drives growth schedules). public int[] Trunk => [.. _trunk.Select(l => l.Weight.Cols)]; + public string Describe() => $"trunk [{string.Join(",", Trunk)}]"; + /// Batched forward pass (autograd-recorded): raw policy logits [B, actions] + value [B, 1]. public (Tensor Logits, Tensor Value) Forward(Tensor observations) { diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorConv.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorConv.cs new file mode 100644 index 0000000..c8579e5 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorConv.cs @@ -0,0 +1,165 @@ +namespace MintPlayer.AI.ReinforcementLearning.Core.Numerics; + +// 2-D convolution as an autograd op (PLAN M42.1). Implemented the standard im2col way: gather each output +// position's receptive field into a row of a [M, inC·kH·kW] matrix, then a single GEMM against the weight +// matrix produces all output channels at once. The heavy multiply therefore runs through Backend.Current +// (GPU-routable), and the reverse pass is just the two transposed GEMMs the backend already provides plus a +// col2im scatter — so a spatial conv net needs NO new backend/kernel, only this data-movement wrapper. +// +// Layout convention: an NCHW image is carried as a rank-2 tensor [N, C·H·W] (row = image, col = c*H*W + h*W + w), +// so the op never produces a rank-4 tensor and never trips the rank-2 assertions the other ops enforce. The +// weight is [inC·kH·kW, outC] and the bias is [outC]. +public sealed partial class Tensor +{ + /// + /// 2-D convolution over an NCHW input carried as [N, inC·inH·inW], producing [N, outC·outH·outW] where + /// outH = (inH + 2·pad − kH)/stride + 1 (outW likewise). is [inC·kH·kW, outC]; + /// is [outC]. Autograd-recorded: the backward pass yields dInput (col2im of dCols), + /// dWeight (colsᵀ·dOut) and dBias (Σ dOut). + /// + public Tensor Conv2D(Tensor weight, Tensor bias, + int inC, int inH, int inW, int outC, int kH, int kW, int stride, int pad) + { + CheckRank2(this); + int n = Rows; + int kk = inC * kH * kW; + if (Cols != inC * inH * inW) + throw new ArgumentException($"Conv2D input cols {Cols} != inC·inH·inW = {inC * inH * inW}."); + if (weight.Rows != kk || weight.Cols != outC) + throw new ArgumentException($"Conv2D weight must be [{kk},{outC}], got [{weight.Rows},{weight.Cols}]."); + if (bias.Length != outC) + throw new ArgumentException($"Conv2D bias length {bias.Length} != outC {outC}."); + if (stride < 1 || pad < 0) throw new ArgumentException("Conv2D needs stride ≥ 1 and pad ≥ 0."); + + int outH = (inH + 2 * pad - kH) / stride + 1; + int outW = (inW + 2 * pad - kW) / stride + 1; + if (outH <= 0 || outW <= 0) + throw new ArgumentException("Conv2D output size is non-positive; check kernel/stride/pad."); + int m = n * outH * outW; + + // im2col → [M, kk], then GEMM against weight [kk, outC] → [M, outC]. + var cols = new float[m * kk]; + Im2Col(Data, cols, n, inC, inH, inW, kH, kW, stride, pad, outH, outW); + var outMat = new float[m * outC]; + Backend.Current.Gemm(cols, weight.Data, outMat, m, kk, outC); + + // Permute [M=n·oH·oW, outC] → [N, outC·oH·oW] and add the per-channel bias. + var data = new float[n * outC * outH * outW]; + ScatterMOutCToNCHW(outMat, bias.Data, data, n, outC, outH, outW); + + return MakeResult(data, [n, outC * outH * outW], [this, weight, bias], result => () => + { + // dOut [N, outC·oH·oW] → dOutMat [M, outC]. + var dOutMat = new float[m * outC]; + GatherNCHWToMOutC(result.Grad!, dOutMat, n, outC, outH, outW); + + if (weight.NeedsGrad) + { + weight.EnsureGrad(); + Backend.Current.GemmTransposeA(cols, dOutMat, weight.Grad!, m, kk, outC); // dW[kk,outC] += colsᵀ·dOut + } + if (bias.NeedsGrad) + { + bias.EnsureGrad(); + var db = bias.Grad!; + for (int i = 0; i < m; i++) + for (int oc = 0; oc < outC; oc++) db[oc] += dOutMat[i * outC + oc]; + } + if (NeedsGrad) + { + EnsureGrad(); + var dCols = new float[m * kk]; + Backend.Current.GemmTransposeB(dOutMat, weight.Data, dCols, m, kk, outC); // dCols[M,kk] += dOut·Wᵀ + Col2Im(dCols, Grad!, n, inC, inH, inW, kH, kW, stride, pad, outH, outW); // scatter-add into dInput + } + }); + } + + // Gather each output position's receptive field into a matrix row: cols[m, (c·kH+kh)·kW+kw] = x[n,c,ih,iw] + // (0 where the padded window falls outside the image). m = (n·oH+oh)·oW+ow. + private static void Im2Col(ReadOnlySpan x, Span cols, + int n, int inC, int inH, int inW, int kH, int kW, int stride, int pad, int outH, int outW) + { + int kk = inC * kH * kW; + for (int img = 0; img < n; img++) + for (int oh = 0; oh < outH; oh++) + for (int ow = 0; ow < outW; ow++) + { + int row = ((img * outH) + oh) * outW + ow; + int colBase = row * kk; + for (int c = 0; c < inC; c++) + for (int kh = 0; kh < kH; kh++) + { + int ih = oh * stride - pad + kh; + for (int kw = 0; kw < kW; kw++) + { + int iw = ow * stride - pad + kw; + int kcol = (c * kH + kh) * kW + kw; + cols[colBase + kcol] = (ih >= 0 && ih < inH && iw >= 0 && iw < inW) + ? x[((img * inC + c) * inH + ih) * inW + iw] + : 0f; + } + } + } + } + + // The transpose of Im2Col: scatter-ADD each matrix row back to the input positions it was gathered from + // (overlapping windows accumulate). Adds into dInput (does not overwrite). + private static void Col2Im(ReadOnlySpan cols, Span dInput, + int n, int inC, int inH, int inW, int kH, int kW, int stride, int pad, int outH, int outW) + { + int kk = inC * kH * kW; + for (int img = 0; img < n; img++) + for (int oh = 0; oh < outH; oh++) + for (int ow = 0; ow < outW; ow++) + { + int row = ((img * outH) + oh) * outW + ow; + int colBase = row * kk; + for (int c = 0; c < inC; c++) + for (int kh = 0; kh < kH; kh++) + { + int ih = oh * stride - pad + kh; + if (ih < 0 || ih >= inH) continue; + for (int kw = 0; kw < kW; kw++) + { + int iw = ow * stride - pad + kw; + if (iw < 0 || iw >= inW) continue; + int kcol = (c * kH + kh) * kW + kw; + dInput[((img * inC + c) * inH + ih) * inW + iw] += cols[colBase + kcol]; + } + } + } + } + + // [M=n·oH·oW, outC] → [N, outC·oH·oW] (+ per-channel bias). out[n,oc,oh,ow] = mat[(n·oH+oh)·oW+ow, oc] + bias[oc]. + private static void ScatterMOutCToNCHW(ReadOnlySpan mat, ReadOnlySpan bias, Span outp, + int n, int outC, int outH, int outW) + { + int hw = outH * outW; + for (int img = 0; img < n; img++) + for (int oh = 0; oh < outH; oh++) + for (int ow = 0; ow < outW; ow++) + { + int row = ((img * outH) + oh) * outW + ow; + int sp = oh * outW + ow; + for (int oc = 0; oc < outC; oc++) + outp[(img * outC + oc) * hw + sp] = mat[row * outC + oc] + bias[oc]; + } + } + + // The inverse index map: [N, outC·oH·oW] → [M, outC]. mat[(n·oH+oh)·oW+ow, oc] = grad[n,oc,oh,ow]. + private static void GatherNCHWToMOutC(ReadOnlySpan nchw, Span mat, + int n, int outC, int outH, int outW) + { + int hw = outH * outW; + for (int img = 0; img < n; img++) + for (int oh = 0; oh < outH; oh++) + for (int ow = 0; ow < outW; ow++) + { + int row = ((img * outH) + oh) * outW + ow; + int sp = oh * outW + ow; + for (int oc = 0; oc < outC; oc++) + mat[row * outC + oc] = nchw[(img * outC + oc) * hw + sp]; + } + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IMaterialScore.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IMaterialScore.cs new file mode 100644 index 0000000..ca02640 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IMaterialScore.cs @@ -0,0 +1,17 @@ +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// +/// An OPTIONAL capability an may also implement to expose a dense, per-position +/// material/score signal (from the side-to-move's perspective). Self-play uses it to shape the value target so +/// a still-weak net gets a gradient on every capture — not just the sparse win/loss/draw outcome, which for a net +/// that can't yet force mate is ~always a draw (→ 0) and teaches nothing. It also gives the difficulty ladder a +/// non-saturating strength metric (two "drawing" nets are separable by average material). Games with no material +/// notion simply don't implement it, and self-play falls back to the pure outcome. +/// +/// The game state type. +public interface IMaterialScore +{ + /// The side-to-move's material advantage (own material − opponent's), in the game's natural units + /// (chess: pawns). Positive = the side to move is ahead. + float MaterialAdvantage(TState state); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs new file mode 100644 index 0000000..c6aabc0 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs @@ -0,0 +1,56 @@ +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// A terminal game result, stated RELATIVE TO THE SIDE TO MOVE in the queried state. +public enum GameResult +{ + /// Not a terminal position. + Ongoing, + /// The side to move has already won (rare — most games are won on the mover's own move). + Win, + /// The side to move has lost (e.g. it is checkmated / the opponent just completed a line). + Loss, + /// A draw (stalemate, repetition, full board, …). + Draw, +} + +/// +/// A perfect-information, two-player, zero-sum, deterministic game — the shared shape that MCTS +/// () and a self-play trainer both consume. It is deliberately distinct from +/// IEnvironment (which has a single-agent reward/episode loop) and from +/// (whose single-goal IsGoal and always-valid ActionCount can't express side-to-move, +/// win/loss/draw, or per-state legal moves). +/// +/// Conventions the implementer MUST honour: returns a FRESH successor and leaves the input +/// state untouched (search reuses one state across many candidate moves), and applying a move FLIPS the side +/// to move. and are always from the perspective of the side to +/// move in the given state, so the network sees one canonical "me vs. them" view regardless of whose turn it is. +/// +/// +/// The immutable-from-the-caller's-view game state (a position). +public interface IZeroSumGame +{ + /// The fixed action-index space — equals the policy head width of the net that plays this game. + /// A given state's legal subset is ; every legal move is an index in [0, PolicySize). + int PolicySize { get; } + + /// The length of the observation writes (the net's input width). + int ObservationSize { get; } + + /// The start position. is for games with a randomized setup (chess/Connect-4 ignore it). + TState Root(ulong? seed = null); + + /// The legal action indices in (a subset of [0, PolicySize)). Never empty for a + /// non-terminal state. + IReadOnlyList LegalMoves(TState state); + + /// The successor after playing — a fresh state, with the side to move flipped. + /// is not mutated. must be one of . + TState Apply(TState state, int move); + + /// The terminal result for the side to move in , or . + GameResult Result(TState state); + + /// Writes the side-to-move-relative observation of into + /// (length ). + void WriteObservation(TState state, Span destination); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs new file mode 100644 index 0000000..f85f39f --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs @@ -0,0 +1,178 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// +/// AlphaZero-style PUCT Monte-Carlo Tree Search over an , guided by a neural +/// leaf evaluator (policy priors + a value in [-1,1]). It composes over the game seam + an +/// delegate exactly as composes over + a +/// cost-to-go delegate, so Core carries no game or NN dependency. +/// +/// Each simulation walks the tree by maximizing Q + U (mean action value + a prior-weighted exploration +/// bonus), expands one new leaf, evaluates it with the net, and backs the value up the path negating it every +/// ply (zero-sum: a position good for the child's mover is bad for the parent's). At the root, Dirichlet noise is +/// mixed into the priors so self-play keeps exploring. returns the root's visit-count +/// distribution — the policy target π for training, from which the caller samples (early game) or takes the argmax. +/// +/// +public static class Mcts +{ + /// Tree simulations per move (more = stronger, slower). + /// PUCT exploration constant. + /// Concentration of the root-noise Dirichlet (chess ≈ 0.3; smaller = spikier). + /// Fraction of the root prior replaced by Dirichlet noise (0 disables). + public sealed record Config(int Simulations = 200, float Cpuct = 1.25f, float DirichletAlpha = 0.3f, float RootNoiseFrac = 0.25f); + + /// Evaluates a leaf: a probability distribution over PolicySize (illegal moves ≈ 0; the search + /// renormalizes over the legal set) and the position value in [-1,1] from the side-to-move's perspective. Backed + /// by a PolicyValueNet.Forward under NoGrad, softmaxed over legal moves + tanh value. + public delegate (float[] Priors, float Value) Evaluate(TState state); + + private sealed class Node + { + public required int[] Moves; // legal action indices in this state + public float[] P = []; // prior per move (aligned to Moves) + public int[] N = []; // visit count per move + public float[] W = []; // summed value per move, from THIS node's mover perspective + public Node?[] Children = []; + public bool Expanded; + public bool Terminal; + public float TerminalValue; // set when Terminal: +1 win / -1 loss / 0 draw, mover-relative + } + + /// Runs .Simulations from and returns the root + /// visit-count distribution over game.PolicySize (sums to 1 over the legal moves; 0 elsewhere). Falls back + /// to the raw priors only if no simulation recorded a visit (Simulations == 0). + public static float[] Search(IZeroSumGame game, TState rootState, + Evaluate evaluate, Config config, Xoshiro256StarStar rng) + { + var root = NewNode(game, rootState); + if (!root.Terminal) + { + ExpandLeaf(root, evaluate, rootState); + AddRootNoise(root, config, rng); + for (int sim = 0; sim < config.Simulations; sim++) + Simulate(root, game, rootState, evaluate, config); + } + + var pi = new float[game.PolicySize]; + int total = 0; + for (int i = 0; i < root.Moves.Length; i++) total += root.N[i]; + if (total == 0) + { + for (int i = 0; i < root.Moves.Length; i++) pi[root.Moves[i]] = root.P[i]; + return pi; + } + for (int i = 0; i < root.Moves.Length; i++) pi[root.Moves[i]] = root.N[i] / (float)total; + return pi; + } + + // Returns the value of 'state' from the perspective of its side to move. + private static float Simulate(Node node, IZeroSumGame game, TState state, + Evaluate evaluate, Config config) + { + if (node.Terminal) return node.TerminalValue; + if (!node.Expanded) return ExpandLeaf(node, evaluate, state); + + int edge = SelectChild(node, config); + var childState = game.Apply(state, node.Moves[edge]); + node.Children[edge] ??= NewNode(game, childState); + float value = -Simulate(node.Children[edge]!, game, childState, evaluate, config); // flip: child mover's value + node.N[edge]++; + node.W[edge] += value; + return value; + } + + private static int SelectChild(Node node, Config config) + { + int sumN = 0; + for (int i = 0; i < node.N.Length; i++) sumN += node.N[i]; + float sqrtSum = MathF.Sqrt(sumN); + + int best = 0; + float bestScore = float.NegativeInfinity; + for (int i = 0; i < node.Moves.Length; i++) + { + float q = node.N[i] > 0 ? node.W[i] / node.N[i] : 0f; + float u = config.Cpuct * node.P[i] * sqrtSum / (1 + node.N[i]); + float score = q + u; + if (score > bestScore) { bestScore = score; best = i; } + } + return best; + } + + private static Node NewNode(IZeroSumGame game, TState state) + { + var result = game.Result(state); + if (result != GameResult.Ongoing) + return new Node { Moves = [], Terminal = true, TerminalValue = result switch { GameResult.Win => 1f, GameResult.Loss => -1f, _ => 0f } }; + return new Node { Moves = [.. game.LegalMoves(state)] }; + } + + // Evaluate the leaf, seed per-move priors (masked to legal + renormalized), and return the net's value estimate. + private static float ExpandLeaf(Node node, Evaluate evaluate, TState state) + { + var (priors, value) = evaluate(state); + int k = node.Moves.Length; + node.P = new float[k]; + node.N = new int[k]; + node.W = new float[k]; + node.Children = new Node?[k]; + + float sum = 0f; + for (int i = 0; i < k; i++) { node.P[i] = MathF.Max(priors[node.Moves[i]], 0f); sum += node.P[i]; } + if (sum > 0f) { for (int i = 0; i < k; i++) node.P[i] /= sum; } + else { for (int i = 0; i < k; i++) node.P[i] = 1f / k; } // net gave no mass to legal moves → uniform + + node.Expanded = true; + return value; + } + + private static void AddRootNoise(Node root, Config config, Xoshiro256StarStar rng) + { + if (config.RootNoiseFrac <= 0f || root.Moves.Length == 0) return; + var noise = SampleDirichlet(root.Moves.Length, config.DirichletAlpha, rng); + float frac = config.RootNoiseFrac; + for (int i = 0; i < root.P.Length; i++) root.P[i] = (1f - frac) * root.P[i] + frac * noise[i]; + } + + // ── Dirichlet sampling (Gamma via Marsaglia–Tsang; normals via Box–Muller), on the game's own RNG stream ── + private static float[] SampleDirichlet(int k, double alpha, Xoshiro256StarStar rng) + { + var g = new double[k]; + double sum = 0; + for (int i = 0; i < k; i++) { g[i] = SampleGamma(alpha, rng); sum += g[i]; } + var d = new float[k]; + if (sum <= 0) { for (int i = 0; i < k; i++) d[i] = 1f / k; return d; } + for (int i = 0; i < k; i++) d[i] = (float)(g[i] / sum); + return d; + } + + private static double SampleGamma(double alpha, Xoshiro256StarStar rng) + { + if (alpha < 1.0) + { + double u = rng.NextDouble(); + return SampleGamma(alpha + 1.0, rng) * Math.Pow(u <= 0 ? double.Epsilon : u, 1.0 / alpha); + } + double d = alpha - 1.0 / 3.0; + double c = 1.0 / Math.Sqrt(9.0 * d); + while (true) + { + double x = NextNormal(rng); + double v = 1 + c * x; + if (v <= 0) continue; + v = v * v * v; + double u2 = rng.NextDouble(); + if (u2 < 1 - 0.0331 * x * x * x * x) return d * v; + if (Math.Log(u2 <= 0 ? double.Epsilon : u2) < 0.5 * x * x + d * (1 - v + Math.Log(v))) return d * v; + } + } + + private static double NextNormal(Xoshiro256StarStar rng) + { + double u1 = rng.NextDouble(); + double u2 = rng.NextDouble(); + return Math.Sqrt(-2.0 * Math.Log(u1 <= 0 ? double.Epsilon : u1)) * Math.Cos(2.0 * Math.PI * u2); + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DeterministicParallel.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DeterministicParallel.cs new file mode 100644 index 0000000..3cdd34d --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/DeterministicParallel.cs @@ -0,0 +1,81 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Training; + +/// +/// Generates a batch of independent items — one per index in [0, count) — with a determinism guarantee: +/// the result array is bitwise-identical whether produced in parallel or sequentially, and at any degree of +/// parallelism. This is the reusable form of the data-generation loop the cube campaigns hand-roll and that +/// self-play needs: episodes/games/labeled-samples are embarrassingly parallel, but a seeded run must stay +/// reproducible regardless of how many cores happen to run it. +/// +/// It holds the same way Core's other parallel primitives do (: per-unit RNG; +/// the backend GEMM: disjoint output rows) — each item's randomness is a pure function of its global index +/// (never execution order), and each item writes only its own result slot, so there is no shared mutable state and +/// no reduction. The caller supplies a + a stream index so generation stays on its own +/// RNG stream, disjoint from the trainer's other streams (init, eval, …), exactly as the rest of the codebase fans +/// seeds out. +/// +/// +public static class DeterministicParallel +{ + /// + /// Runs for each local index in [0, count), each with its OWN RNG derived + /// from (, , + localIndex), + /// and returns the results in ascending index order. Because a given global index always derives the same RNG + /// and writes the same slot, the output does not depend on or on the worker count — + /// callers get free multi-core scaling with zero effect on a seeded run's outcome. + /// + /// Number of items to generate (0 ⇒ empty array). + /// The run's master seed fan-out; the per-item base seed is seeds.Derive(stream). + /// RNG stream index for this generation phase (see ), keeping it + /// disjoint from other streams. + /// Global index of the first item — advance it across calls (e.g. total games so far) + /// so successive batches draw non-overlapping, non-repeating RNGs. + /// Produces item i from its derived RNG. MUST be pure w.r.t. that RNG and any + /// read-only captured state (a shared read-only model snapshot is fine); it must not touch shared mutable state, + /// or the determinism guarantee is lost. + /// Spread work across cores (bitwise-identical to the sequential path). + /// Optional cap on the degree of parallelism (defaults to the runtime's choice). + public static TItem[] Generate( + int count, SeedSequence seeds, int stream, long baseIndex, + Func makeItem, bool parallel, int? maxDop = null) + { + ArgumentNullException.ThrowIfNull(seeds); + return Generate(count, seeds.Derive(stream), baseIndex, makeItem, parallel, maxDop); + } + + /// + /// As above, but for callers that manage their own base seed rather than fanning out a + /// — e.g. a data-gen loop keyed on a per-round seed. Item i's RNG is + /// Xoshiro( + ( + i)·φ), so passing + /// (baseSeed: roundSeed, baseIndex: 1) reproduces the common roundSeed + φ·(worker+1) per-worker + /// seeding exactly. + /// + public static TItem[] Generate( + int count, ulong baseSeed, long baseIndex, + Func makeItem, bool parallel, int? maxDop = null) + { + ArgumentOutOfRangeException.ThrowIfNegative(count); + ArgumentNullException.ThrowIfNull(makeItem); + + var results = new TItem[count]; + if (parallel && count > 1) + { + var options = new ParallelOptions(); + if (maxDop is int dop) options.MaxDegreeOfParallelism = dop; + Parallel.For(0, count, options, i => results[i] = makeItem(i, DeriveRng(baseSeed, baseIndex + i))); + } + else + { + for (int i = 0; i < count; i++) + results[i] = makeItem(i, DeriveRng(baseSeed, baseIndex + i)); + } + return results; + } + + /// Per-item RNG: the golden-ratio index stride the codebase uses everywhere (VectorEnv.Reset), + /// whose xoshiro seeding runs each seed through SplitMix64 — so adjacent indices give decorrelated streams. + private static Xoshiro256StarStar DeriveRng(ulong baseSeed, long globalIndex) + => new(unchecked(baseSeed + (ulong)globalIndex * 0x9E3779B97F4A7C15UL)); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs new file mode 100644 index 0000000..3f899fd --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs @@ -0,0 +1,96 @@ +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// Piece type (colour-agnostic). = empty square. +public enum PieceType : byte { None = 0, Pawn = 1, Knight = 2, Bishop = 3, Rook = 4, Queen = 5, King = 6 } + +/// A move: from/to squares (0..63, = rank*8 + file, a1 = 0), plus the promotion piece for a pawn reaching +/// the last rank ( otherwise). Castling, en passant, and double-push are inferred from +/// the board when the move is made, so they need no extra flag here. +public readonly record struct ChessMove(byte From, byte To, PieceType Promotion = PieceType.None); + +/// +/// A chess position — the public, ergonomic view (mailbox , side to move, castling rights, +/// en-passant target, halfmove clock). This is a thin facade over the single-source engine transpiled from +/// polyglot/chess_solver.pg (PgChessState) — the same source the browser client runs. The board is +/// immutable: every move produces a fresh state. Squares are 0..63 (file = sq % 8, rank = sq / 8); White moves +/// toward higher ranks. Cells: 0 = empty, +1..+6 = White P,N,B,R,Q,K, −1..−6 = Black (sign = colour). +/// Threefold-repetition is intentionally not modelled (see PLAN M39); the 50-move rule and the self-play +/// ply cap bound looping games. +/// +public sealed class ChessState +{ + internal readonly PgChessState Core; + private sbyte[]? _squares; + + public const byte CastleWK = 1, CastleWQ = 2, CastleBK = 4, CastleBQ = 8; + + internal ChessState(PgChessState core) => Core = core; + + public ChessState(sbyte[] squares, bool whiteToMove, byte castling, sbyte enPassant, byte halfmoveClock) + { + var cells = new List(64); + for (int i = 0; i < 64; i++) cells.Add(squares[i]); + Core = new PgChessState(cells, whiteToMove, castling, enPassant, halfmoveClock); + } + + /// The 64-cell mailbox (a cached snapshot of the core board — read-only). + public sbyte[] Squares + { + get + { + if (_squares is null) + { + _squares = new sbyte[64]; + for (int i = 0; i < 64; i++) _squares[i] = (sbyte)Core.squares[i]; + } + return _squares; + } + } + + public bool WhiteToMove => Core.whiteToMove; + public byte Castling => (byte)Core.castling; // bit 0 = White O-O, 1 = White O-O-O, 2 = Black O-O, 3 = Black O-O-O + public sbyte EnPassant => (sbyte)Core.enPassant; + public byte HalfmoveClock => (byte)Core.halfmoveClock; + + public static ChessState StartPosition() => ChessFen.Parse(ChessFen.StartFen); + + // Move ↔ generated-move mapping (promotion piece codes coincide with PieceType's byte values, so the cast is + // a no-op renaming). Shared by ChessRules and ChessMoveEncoding — the two facades over the single source. + internal static PgChessMove ToPg(ChessMove m) => new(m.From, m.To, (int)m.Promotion); + internal static ChessMove FromPg(PgChessMove m) => new((byte)m.from, (byte)m.to, (PieceType)m.promotion); +} + +/// +/// The chess rules — a public facade over the single-source engine (chess_solver.pg → PgChessState): +/// legal move generation (castling, en passant, promotion, pins/checks), make-move, attack detection, and terminal +/// detection (checkmate, stalemate, 50-move, insufficient material). Correctness is pinned by perft, which +/// recurses entirely inside the generated core (no per-node marshalling). +/// +public static class ChessRules +{ + public static bool IsSquareAttacked(ChessState s, int square, bool byWhite) => s.Core.isSquareAttacked(square, byWhite); + + public static int KingSquare(ChessState s, bool white) => s.Core.kingSquare(white); + + public static bool InCheck(ChessState s, bool white) => s.Core.inCheck(white); + + /// All fully-legal moves for the side to move (own king not left in check). + public static List LegalMoves(ChessState s) + { + var core = s.Core.legalMoves(); + var moves = new List(core.Count); + foreach (var m in core) moves.Add(ChessState.FromPg(m)); + return moves; + } + + /// The position after (a fresh state; is unchanged). + public static ChessState MakeMove(ChessState s, ChessMove move) => new(s.Core.makeMove(ChessState.ToPg(move))); + + public static bool IsFiftyMove(ChessState s) => s.Core.isFiftyMove(); + + public static bool IsInsufficientMaterial(ChessState s) => s.Core.isInsufficientMaterial(); + + /// Perft: the number of leaf nodes at — the movegen correctness oracle. The + /// count is produced entirely inside the generated engine (i32 there; widened to long here). + public static long Perft(ChessState s, int depth) => s.Core.perft(depth); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs new file mode 100644 index 0000000..23599ba --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs @@ -0,0 +1,101 @@ +using System.Text; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// Forsyth–Edwards Notation (FEN) ↔ . Used to set up standard test positions (perft) and to +/// render a position for debugging. Only the fields the engine tracks are parsed (board, side, castling, en passant, +/// halfmove clock); the fullmove number is ignored on read and written as 1. +/// +public static class ChessFen +{ + public const string StartFen = "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"; + + public static ChessState Parse(string fen) + { + string[] parts = fen.Split(' '); + var squares = new sbyte[64]; + + string[] ranks = parts[0].Split('/'); // rank 8 first + for (int i = 0; i < 8; i++) + { + int rank = 7 - i; // ranks[0] is the 8th rank (rank index 7) + int file = 0; + foreach (char c in ranks[i]) + { + if (char.IsDigit(c)) { file += c - '0'; continue; } + squares[rank * 8 + file] = PieceOf(c); + file++; + } + } + + bool whiteToMove = parts[1] == "w"; + byte castling = 0; + if (parts[2] != "-") + foreach (char c in parts[2]) + castling |= c switch + { + 'K' => ChessState.CastleWK, + 'Q' => ChessState.CastleWQ, + 'k' => ChessState.CastleBK, + 'q' => ChessState.CastleBQ, + _ => (byte)0, + }; + + sbyte enPassant = parts[3] == "-" ? (sbyte)-1 : (sbyte)SquareOf(parts[3]); + byte halfmove = parts.Length > 4 && byte.TryParse(parts[4], out byte hm) ? hm : (byte)0; + return new ChessState(squares, whiteToMove, castling, enPassant, halfmove); + } + + public static string ToFen(ChessState s) + { + var sb = new StringBuilder(); + for (int rank = 7; rank >= 0; rank--) + { + int empty = 0; + for (int file = 0; file < 8; file++) + { + sbyte piece = s.Squares[rank * 8 + file]; + if (piece == 0) { empty++; continue; } + if (empty > 0) { sb.Append(empty); empty = 0; } + sb.Append(CharOf(piece)); + } + if (empty > 0) sb.Append(empty); + if (rank > 0) sb.Append('/'); + } + sb.Append(s.WhiteToMove ? " w " : " b "); + string rights = string.Concat( + (s.Castling & ChessState.CastleWK) != 0 ? "K" : "", + (s.Castling & ChessState.CastleWQ) != 0 ? "Q" : "", + (s.Castling & ChessState.CastleBK) != 0 ? "k" : "", + (s.Castling & ChessState.CastleBQ) != 0 ? "q" : ""); + sb.Append(rights.Length == 0 ? "-" : rights); + sb.Append(' ').Append(s.EnPassant < 0 ? "-" : SquareName(s.EnPassant)); + sb.Append(' ').Append(s.HalfmoveClock).Append(" 1"); + return sb.ToString(); + } + + private static sbyte PieceOf(char c) + { + sbyte type = char.ToLowerInvariant(c) switch + { + 'p' => 1, 'n' => 2, 'b' => 3, 'r' => 4, 'q' => 5, 'k' => 6, + _ => throw new FormatException($"Bad FEN piece '{c}'."), + }; + return char.IsUpper(c) ? type : (sbyte)-type; + } + + private static char CharOf(sbyte piece) + { + char c = (PieceType)Math.Abs(piece) switch + { + PieceType.Pawn => 'p', PieceType.Knight => 'n', PieceType.Bishop => 'b', + PieceType.Rook => 'r', PieceType.Queen => 'q', PieceType.King => 'k', + _ => '?', + }; + return piece > 0 ? char.ToUpperInvariant(c) : c; + } + + private static int SquareOf(string s) => (s[1] - '1') * 8 + (s[0] - 'a'); + private static string SquareName(int sq) => $"{(char)('a' + (sq & 7))}{(char)('1' + (sq >> 3))}"; +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs new file mode 100644 index 0000000..2f9da91 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs @@ -0,0 +1,74 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// Chess as an — the headline consumer of the self-play stack (PLAN M39.2). It +/// adapts the perft-verified engine to the seam: the action space is the 4672-index +/// , and the observation is an 18-plane board stack (12 piece planes + side-to-move + +/// 4 castling-rights + en passant), flattened. MCTS, the two-headed net, and the self-play campaign are all reused +/// unchanged from M39.1. (Board geometry is absolute — the net learns both colours via the side-to-move plane; +/// perspective canonicalization is a later efficiency lever, PLAN M39.3.) +/// +public sealed class ChessGame : IZeroSumGame, IMaterialScore +{ + private const int Planes = 18; + + // Standard relative piece values, indexed by |piece| (none, P, N, B, R, Q, K). The king isn't "captured" + // (reaching it is checkmate — the ±1 game outcome), so it scores 0 and cancels between the two sides. + private static readonly int[] PieceValues = [0, 1, 3, 3, 5, 9, 0]; + + public int PolicySize => ChessMoveEncoding.Size; // 4672 + public int ObservationSize => Planes * 64; // 1152 + + /// The side-to-move's material advantage in pawns (its pieces − the opponent's). Dense reward signal + /// for self-play shaping + the difficulty ladder's strength metric (). + public float MaterialAdvantage(ChessState state) + { + int white = 0, black = 0; + foreach (sbyte p in state.Squares) + { + int v = PieceValues[Math.Abs(p)]; + if (p > 0) white += v; else if (p < 0) black += v; + } + int diff = white - black; + return state.WhiteToMove ? diff : -diff; + } + + public ChessState Root(ulong? seed = null) => ChessState.StartPosition(); + + // The seam methods delegate to the single-source core so training and the browser client share one + // implementation of "legal indices / apply an index / terminal result". + public IReadOnlyList LegalMoves(ChessState state) => state.Core.legalMoveIndices(); + + public ChessState Apply(ChessState state, int move) => new(state.Core.applyIndex(move)); + + public GameResult Result(ChessState state) => state.Core.result() switch + { + 1 => GameResult.Loss, // side to move is checkmated + 2 => GameResult.Draw, // stalemate / 50-move / insufficient material + _ => GameResult.Ongoing, + }; + + public void WriteObservation(ChessState state, Span destination) + { + destination.Clear(); + // Piece planes 0..5 = White P,N,B,R,Q,K ; 6..11 = Black. Plane p, square sq → index p*64 + sq. + for (int sq = 0; sq < 64; sq++) + { + sbyte piece = state.Squares[sq]; + if (piece == 0) continue; + int type = Math.Abs(piece) - 1; // 0..5 + int plane = piece > 0 ? type : 6 + type; // white vs black block + destination[plane * 64 + sq] = 1f; + } + if (state.WhiteToMove) Fill(destination, 12); + if ((state.Castling & ChessState.CastleWK) != 0) Fill(destination, 13); + if ((state.Castling & ChessState.CastleWQ) != 0) Fill(destination, 14); + if ((state.Castling & ChessState.CastleBK) != 0) Fill(destination, 15); + if ((state.Castling & ChessState.CastleBQ) != 0) Fill(destination, 16); + if (state.EnPassant >= 0) destination[17 * 64 + state.EnPassant] = 1f; + } + + private static void Fill(Span dest, int plane) => dest.Slice(plane * 64, 64).Fill(1f); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs new file mode 100644 index 0000000..5de0116 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs @@ -0,0 +1,25 @@ +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// The AlphaZero move encoding: a fixed 64 × 73 = 4672 action space (from-square × move-type plane), which is +/// the policy-head width. The 73 planes are 56 "queen" moves (8 directions × 7 distances — this also carries +/// queen-promotions and every sliding/king/pawn move), 8 knight moves, and 9 underpromotions (Knight/Bishop/Rook × +/// {capture-left, straight, capture-right}). Distinct legal moves always map to distinct indices, so the search can +/// tell them apart; queen-promotions ride the queen planes (decoded promotion is inferred as Queen when a pawn +/// reaches the last rank — see ). +/// A thin facade over the single-source encoder (chess_solver.pg → PgChessState.encode/decode), +/// so the browser client and the training seam share one implementation. +/// +public static class ChessMoveEncoding +{ + public const int PlanesPerSquare = 73; + public const int Size = 64 * PlanesPerSquare; // 4672 + + /// The action index (0..4671) for a legal move. Queen-promotions use the queen planes; N/B/R + /// promotions use the underpromotion planes. + public static int Encode(ChessMove move) => PgChessState.encode(ChessState.ToPg(move)); + + /// Decodes an action index to a move. A queen-promotion decodes with — the + /// caller promotes a pawn reaching the last rank to a Queen by default. + public static ChessMove Decode(int index) => ChessState.FromPg(PgChessState.decode(index)); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg new file mode 100644 index 0000000..3d27ae3 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg @@ -0,0 +1,740 @@ +// chess_solver.pg — single-source port of the perft-verified chess engine +// (ChessBoard.cs: ChessState + ChessRules, and ChessMoveEncoding.cs). ONE source, +// transpiled to C# (obj/, build-time, wrapped by the ChessState/ChessRules facades) +// and — from M40.2 — to a committed TypeScript twin the browser runs client-side. +// +// Types are `Pg`-prefixed and stay internal; the hand-written C# facades +// (ChessBoard.cs, ChessMoveEncoding.cs) adapt this i32/List core to the public +// PieceType/ChessMove/sbyte[] API the rest of the SDK and the perft tests consume. +// Perft (the movegen correctness gate) recurses entirely in this core. +// +// Subset choices (mirror the FruitCake precedent): the mailbox board is a List +// of 64 cells (0 empty, +1..+6 White P,N,B,R,Q,K, −1..−6 Black); promotion is an i32 +// piece code (0 = none, 2..5 = N,B,R,Q); castling is an i32 bitmask (WK=1,WQ=2,BK=4, +// BQ=8). No i64 — perft returns i32 (fits every position/depth the tests probe; +// the C# facade widens to long). Ray/slide loops are bounded `for`+`continue` (no +// `while`); delta tables are List<(i32,i32)> iterated with tuple destructuring — every +// construct here is one proven by fruitcake_solver.pg. + +import { List } from "std.collections" +import { Math } from "std.math" + +// A move: from/to squares (0..63 = rank*8 + file) plus the promotion piece code +// (0 = none) for a pawn reaching the last rank. Castling / en passant / double-push +// are inferred from the board when the move is made, so they need no extra flag. +record PgChessMove(from: i32, to: i32, promotion: i32) + +class PgChessState { + const PromoKnight: i32 = 2 + const PromoBishop: i32 = 3 + const PromoRook: i32 = 4 + const PromoQueen: i32 = 5 + + const CastleWK: i32 = 1 + const CastleWQ: i32 = 2 + const CastleBK: i32 = 4 + const CastleBQ: i32 = 8 + + const Size: i32 = 4672 // 64 × 73 AlphaZero action space + const PlanesPerSquare: i32 = 73 + + var squares: List // length 64 + var whiteToMove: bool + var castling: i32 + var enPassant: i32 // target square a pawn could capture onto, or -1 + var halfmoveClock: i32 + + init(squares: List, whiteToMove: bool, castling: i32, enPassant: i32, halfmoveClock: i32) { + this.squares = squares + this.whiteToMove = whiteToMove + this.castling = castling + this.enPassant = enPassant + this.halfmoveClock = halfmoveClock + } + + // ── small helpers (std.math covers abs/max/min type-preserving on i32; only sign is missing) ─ + static fn fileOf(sq: i32): i32 => sq % 8 + static fn rankOf(sq: i32): i32 => sq / 8 + static fn isign(x: i32): i32 => if x > 0 { 1 } else { if x < 0 { -1 } else { 0 } } + static fn onBoard(f: i32, r: i32): bool => f >= 0 && f < 8 && r >= 0 && r < 8 + static fn isWhite(piece: i32): bool => piece > 0 + static fn pieceType(piece: i32): i32 => if piece < 0 { -piece } else { piece } + // Clear the given bit(s) of a mask without a bitwise-not: drop the parts that ARE in `bits`. + static fn clearBits(mask: i32, bits: i32): i32 => mask - (mask & bits) + + // ── delta tables (built fresh; iterated with tuple destructuring) ───────────────────────── + static fn knightDeltas(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 2)); d.add((2, 1)); d.add((2, -1)); d.add((1, -2)) + d.add((-1, -2)); d.add((-2, -1)); d.add((-2, 1)); d.add((-1, 2)) + return d + } + static fn kingDeltas(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 0)); d.add((1, 1)); d.add((0, 1)); d.add((-1, 1)) + d.add((-1, 0)); d.add((-1, -1)); d.add((0, -1)); d.add((1, -1)) + return d + } + static fn bishopDirs(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 1)); d.add((-1, 1)); d.add((1, -1)); d.add((-1, -1)) + return d + } + static fn rookDirs(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 0)); d.add((-1, 0)); d.add((0, 1)); d.add((0, -1)) + return d + } + // Encoding tables (AlphaZero plane order — shared by encode and decode). + static fn encDirs(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((0, 1)); d.add((1, 1)); d.add((1, 0)); d.add((1, -1)) + d.add((0, -1)); d.add((-1, -1)); d.add((-1, 0)); d.add((-1, 1)) + return d + } + static fn encKnights(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 2)); d.add((2, 1)); d.add((2, -1)); d.add((1, -2)) + d.add((-1, -2)); d.add((-2, -1)); d.add((-2, 1)); d.add((-1, 2)) + return d + } + + fn clone(): PgChessState { + var sq: List = List() + for v in this.squares { sq.add(v) } + return PgChessState(sq, this.whiteToMove, this.castling, this.enPassant, this.halfmoveClock) + } + + // ── attack detection ────────────────────────────────────────────────────────────────────── + fn isSquareAttacked(square: i32, byWhite: bool): bool { + let tf = fileOf(square) + let tr = rankOf(square) + + // Pawns: an attacking pawn sits one rank behind the target in its forward direction. + let pawnDir = if byWhite { 1 } else { -1 } + let pawn = if byWhite { 1 } else { -1 } + for k in 0..2 { + let df = k * 2 - 1 // -1, +1 + let f = tf + df + let r = tr - pawnDir + if onBoard(f, r) && this.squares[r * 8 + f] == pawn { return true } + } + + let knight = if byWhite { 2 } else { -2 } + let kn = knightDeltas() + for (df, dr) in kn { + let f = tf + df + let r = tr + dr + if onBoard(f, r) && this.squares[r * 8 + f] == knight { return true } + } + + let king = if byWhite { 6 } else { -6 } + let kg = kingDeltas() + for (df, dr) in kg { + let f = tf + df + let r = tr + dr + if onBoard(f, r) && this.squares[r * 8 + f] == king { return true } + } + + if this.rayHits(tf, tr, bishopDirs(), byWhite, 3) { return true } // bishops/queens + if this.rayHits(tf, tr, rookDirs(), byWhite, 4) { return true } // rooks/queens + return false + } + + // A sliding piece of type `slider` (or a queen, code 5) of colour `byWhite` attacks (tf,tr) + // along one of `dirs`. Bounded to 7 steps; `blocked` stops the walk at the first obstruction. + fn rayHits(tf: i32, tr: i32, dirs: List<(i32, i32)>, byWhite: bool, slider: i32): bool { + for (df, dr) in dirs { + var found = false + var blocked = false + for step in 1..8 { + if blocked { continue } + let f = tf + df * step + let r = tr + dr * step + if !onBoard(f, r) { blocked = true; continue } + let piece = this.squares[r * 8 + f] + if piece != 0 { + if isWhite(piece) == byWhite { + let t = pieceType(piece) + if t == slider || t == 5 { found = true } + } + blocked = true + } + } + if found { return true } + } + return false + } + + fn kingSquare(white: bool): i32 { + let king = if white { 6 } else { -6 } + for sq in 0..64 { if this.squares[sq] == king { return sq } } + return -1 + } + + fn inCheck(white: bool): bool => this.isSquareAttacked(this.kingSquare(white), !white) + + // ── move generation ───────────────────────────────────────────────────────────────────── + fn legalMoves(): List { + var legal: List = List() + let pseudo = this.pseudoLegal() + for m in pseudo { + let next = this.makeMove(m) + if !next.inCheck(this.whiteToMove) { legal.add(m) } // the side that just moved must be safe + } + return legal + } + + fn pseudoLegal(): List { + var moves: List = List() + let white = this.whiteToMove + for sq in 0..64 { + let piece = this.squares[sq] + if piece == 0 || isWhite(piece) != white { continue } + let f = fileOf(sq) + let r = rankOf(sq) + let t = pieceType(piece) + if t == 1 { this.pawnMoves(sq, f, r, white, moves) } + else if t == 2 { this.stepMoves(sq, f, r, white, knightDeltas(), moves) } + else if t == 6 { this.stepMoves(sq, f, r, white, kingDeltas(), moves); this.castleMoves(white, moves) } + else if t == 3 { this.slideMoves(sq, f, r, white, bishopDirs(), moves) } + else if t == 4 { this.slideMoves(sq, f, r, white, rookDirs(), moves) } + else if t == 5 { + this.slideMoves(sq, f, r, white, bishopDirs(), moves) + this.slideMoves(sq, f, r, white, rookDirs(), moves) + } + } + return moves + } + + fn stepMoves(sq: i32, f: i32, r: i32, white: bool, deltas: List<(i32, i32)>, moves: List) { + for (df, dr) in deltas { + let nf = f + df + let nr = r + dr + if !onBoard(nf, nr) { continue } + let to = nr * 8 + nf + let target = this.squares[to] + if target == 0 || isWhite(target) != white { moves.add(PgChessMove(sq, to, 0)) } + } + } + + fn slideMoves(sq: i32, f: i32, r: i32, white: bool, dirs: List<(i32, i32)>, moves: List) { + for (df, dr) in dirs { + var blocked = false + for step in 1..8 { + if blocked { continue } + let nf = f + df * step + let nr = r + dr * step + if !onBoard(nf, nr) { blocked = true; continue } + let to = nr * 8 + nf + let target = this.squares[to] + if target == 0 { moves.add(PgChessMove(sq, to, 0)) } + else { + if isWhite(target) != white { moves.add(PgChessMove(sq, to, 0)) } + blocked = true + } + } + } + } + + fn pawnMoves(sq: i32, f: i32, r: i32, white: bool, moves: List) { + let dir = if white { 1 } else { -1 } + let startRank = if white { 1 } else { 6 } + let lastRank = if white { 7 } else { 0 } + + let one = (r + dir) * 8 + f + if this.squares[one] == 0 { + this.addPawn(sq, one, r + dir == lastRank, moves) + if r == startRank { + let two = (r + 2 * dir) * 8 + f + if this.squares[two] == 0 { moves.add(PgChessMove(sq, two, 0)) } + } + } + + for k in 0..2 { + let df = k * 2 - 1 // -1, +1 + let nf = f + df + let nr = r + dir + if !onBoard(nf, nr) { continue } + let to = nr * 8 + nf + let target = this.squares[to] + if target != 0 && isWhite(target) != white { this.addPawn(sq, to, nr == lastRank, moves) } + else if to == this.enPassant { moves.add(PgChessMove(sq, to, 0)) } // en passant onto the empty ep square + } + } + + fn addPawn(from: i32, to: i32, promotion: bool, moves: List) { + if promotion { + moves.add(PgChessMove(from, to, PromoQueen)) + moves.add(PgChessMove(from, to, PromoRook)) + moves.add(PgChessMove(from, to, PromoBishop)) + moves.add(PgChessMove(from, to, PromoKnight)) + } else { + moves.add(PgChessMove(from, to, 0)) + } + } + + fn castleMoves(white: bool, moves: List) { + let rank = if white { 0 } else { 7 } + let kingSq = rank * 8 + 4 + let ownKing = if white { 6 } else { -6 } + if this.squares[kingSq] != ownKing { return } + if this.isSquareAttacked(kingSq, !white) { return } // can't castle out of check + + let kSide = if white { CastleWK } else { CastleBK } + let qSide = if white { CastleWQ } else { CastleBQ } + + // King-side: f,g empty; king doesn't pass through/into attack. + let kEmpty = this.squares[rank * 8 + 5] == 0 && this.squares[rank * 8 + 6] == 0 + let kSafe = !this.isSquareAttacked(rank * 8 + 5, !white) && !this.isSquareAttacked(rank * 8 + 6, !white) + if (this.castling & kSide) != 0 && kEmpty && kSafe { moves.add(PgChessMove(kingSq, rank * 8 + 6, 0)) } + + // Queen-side: b,c,d empty; king passes over d,c (b need not be safe, only empty). + let qEmpty = this.squares[rank * 8 + 1] == 0 && this.squares[rank * 8 + 2] == 0 && this.squares[rank * 8 + 3] == 0 + let qSafe = !this.isSquareAttacked(rank * 8 + 3, !white) && !this.isSquareAttacked(rank * 8 + 2, !white) + if (this.castling & qSide) != 0 && qEmpty && qSafe { moves.add(PgChessMove(kingSq, rank * 8 + 2, 0)) } + } + + // ── make move ─────────────────────────────────────────────────────────────────────────── + // The position after `move` (a fresh state; this one is unchanged). Castling / en passant / + // promotion / double-push are inferred from the board. + fn makeMove(move: PgChessMove): PgChessState { + var b: List = List() + for v in this.squares { b.add(v) } + let white = this.whiteToMove + let piece = b[move.from] + let t = pieceType(piece) + var capture = b[move.to] != 0 + var newEp = -1 + var castling = this.castling + + b[move.from] = 0 + + // En passant capture: the taken pawn sits beside the destination, not on it. + if t == 1 && move.to == this.enPassant && this.enPassant >= 0 { + let capturedSq = move.to + (if white { -8 } else { 8 }) + b[capturedSq] = 0 + capture = true + } + + // Place the piece (promotion swaps in the chosen piece). + if move.promotion != 0 { + b[move.to] = (if white { 1 } else { -1 }) * move.promotion + } else { + b[move.to] = piece + } + + // Double push sets the en-passant target square. + if t == 1 && Math.abs(rankOf(move.to) - rankOf(move.from)) == 2 { + newEp = (move.from + move.to) / 2 + } + + // Castling: move the rook too. + if t == 6 && Math.abs(fileOf(move.to) - fileOf(move.from)) == 2 { + let rank = rankOf(move.from) + if fileOf(move.to) == 6 { b[rank * 8 + 5] = b[rank * 8 + 7]; b[rank * 8 + 7] = 0 } // king-side + else { b[rank * 8 + 3] = b[rank * 8 + 0]; b[rank * 8 + 0] = 0 } // queen-side + } + + // Update castling rights: a king move clears both of its sides; a rook move/capture on a + // home corner clears that right. + if t == 6 { + if white { castling = clearBits(castling, CastleWK + CastleWQ) } + else { castling = clearBits(castling, CastleBK + CastleBQ) } + } + castling = clearBits(castling, cornerRight(move.from)) + castling = clearBits(castling, cornerRight(move.to)) // a rook captured on its home square loses that right too + + let halfmove = if t == 1 || capture { 0 } else { this.halfmoveClock + 1 } + return PgChessState(b, !white, castling, newEp, halfmove) + } + + // The castling-right bit anchored at a rook/king home corner (0 = none). + static fn cornerRight(sq: i32): i32 { + if sq == 0 { return CastleWQ } // a1 rook + if sq == 7 { return CastleWK } // h1 rook + if sq == 56 { return CastleBQ } // a8 rook + if sq == 63 { return CastleBK } // h8 rook + return 0 + } + + // ── terminal detection ────────────────────────────────────────────────────────────────── + fn isFiftyMove(): bool => this.halfmoveClock >= 100 + + fn isInsufficientMaterial(): bool { + var knights = 0 + var bishops = 0 + for p in this.squares { + let t = pieceType(p) + if t == 0 || t == 6 { continue } + else if t == 2 { knights += 1 } + else if t == 3 { bishops += 1 } + else { return false } // a pawn/rook/queen = enough material + } + return knights + bishops <= 1 // K vs K, K+N vs K, K+B vs K (the common draws) + } + + // Terminal result FROM THE SIDE TO MOVE's view: 0 = ongoing, 1 = loss (checkmated), 2 = draw. + fn result(): i32 { + if this.legalMoves().count == 0 { + return if this.inCheck(this.whiteToMove) { 1 } else { 2 } // mate vs stalemate + } + if this.isFiftyMove() || this.isInsufficientMaterial() { return 2 } + return 0 + } + + // Perft: the number of leaf nodes at `depth` — the movegen correctness oracle. i32 suffices + // for every position/depth the tests probe (max ~4.9M); the C# facade widens to long. + fn perft(depth: i32): i32 { + if depth == 0 { return 1 } + let moves = this.legalMoves() + if depth == 1 { return moves.count } + var nodes = 0 + for m in moves { nodes += this.makeMove(m).perft(depth - 1) } + return nodes + } + + // ── seam helpers (index-level API the C# facade / browser consume) ──────────────────────── + fn legalMoveIndices(): List { + var out: List = List() + let moves = this.legalMoves() + for m in moves { out.add(encode(m)) } + return out + } + + // The 18-plane × 64 = 1152 observation the net consumes (single-sourced; the C# facade casts f64→float + // for the float32 net, the browser feeds it directly). Planes: [0–5] White P,N,B,R,Q,K, [6–11] Black, + // [12] side-to-move (all-ones iff white to move), [13–16] castling WK/WQ/BK/BQ (each all-ones iff the + // right is held), [17] en-passant target square (one-hot). Must match ChessGame.WriteObservation exactly. + fn writeObservation(): List { + var obs: List = List() + for i in 0..1152 { obs.add(0.0) } + + for sq in 0..64 { + let piece = this.squares[sq] + if piece == 0 { continue } + let t = pieceType(piece) - 1 // 0..5 + let plane = if piece > 0 { t } else { 6 + t } // white vs black block + obs[plane * 64 + sq] = 1.0 + } + if this.whiteToMove { this.fillPlane(obs, 12) } + if (this.castling & CastleWK) != 0 { this.fillPlane(obs, 13) } + if (this.castling & CastleWQ) != 0 { this.fillPlane(obs, 14) } + if (this.castling & CastleBK) != 0 { this.fillPlane(obs, 15) } + if (this.castling & CastleBQ) != 0 { this.fillPlane(obs, 16) } + if this.enPassant >= 0 { obs[17 * 64 + this.enPassant] = 1.0 } + return obs + } + + fn fillPlane(obs: List, plane: i32) { + for sq in 0..64 { obs[plane * 64 + sq] = 1.0 } + } + + // Apply an encoded action index (a queen-promotion rides the queen planes and decodes with + // promotion 0 → promote a pawn reaching the last rank to a Queen by default). + fn applyIndex(index: i32): PgChessState { + let m = decode(index) + var chosen = m + if m.promotion == 0 && pieceType(this.squares[m.from]) == 1 { + let toRank = rankOf(m.to) + if toRank == 0 || toRank == 7 { chosen = PgChessMove(m.from, m.to, PromoQueen) } + } + return this.makeMove(chosen) + } + + // ── AlphaZero 4672 move encoding (64 × 73 = from-square × plane) ────────────────────────── + // 56 queen planes (8 dir × 7 dist — also carry queen-promotions and king/pawn moves) + 8 + // knight planes + 9 underpromotion planes (N/B/R × {capture-left, straight, capture-right}). + static fn encode(move: PgChessMove): i32 { + let df = fileOf(move.to) - fileOf(move.from) + let dr = rankOf(move.to) - rankOf(move.from) + var plane = 0 + if move.promotion == PromoKnight || move.promotion == PromoBishop || move.promotion == PromoRook { + let pieceIdx = if move.promotion == PromoKnight { 0 } else { if move.promotion == PromoBishop { 1 } else { 2 } } + plane = 64 + pieceIdx * 3 + (df + 1) // df ∈ {-1,0,1} → {capture-left, straight, capture-right} + } else if isKnightMove(df, dr) { + plane = 56 + knightIndex(df, dr) + } else { + let dir = dirIndex(isign(df), isign(dr)) + let dist = Math.max(Math.abs(df), Math.abs(dr)) + plane = dir * 7 + (dist - 1) + } + return move.from * PlanesPerSquare + plane + } + + // Decode an action index to a move. A queen-promotion decodes with promotion 0 — the caller + // promotes a pawn reaching the last rank to a Queen by default (see applyIndex). + static fn decode(index: i32): PgChessMove { + let from = index / PlanesPerSquare + let plane = index % PlanesPerSquare + let f = fileOf(from) + let r = rankOf(from) + + if plane >= 64 { + let p = plane - 64 + let promo = if p / 3 == 0 { PromoKnight } else { if p / 3 == 1 { PromoBishop } else { PromoRook } } + let df = (p % 3) - 1 + let dr = if r == 6 { 1 } else { -1 } // a pawn on the 7th rank promotes upward, on the 2nd downward + return PgChessMove(from, (r + dr) * 8 + (f + df), promo) + } + if plane >= 56 { + let target = plane - 56 + var i = 0 + var rdf = 0 + var rdr = 0 + for (df, dr) in encKnights() { + if i == target { rdf = df; rdr = dr } + i += 1 + } + return PgChessMove(from, (r + rdr) * 8 + (f + rdf), 0) + } + let dirTarget = plane / 7 + let dist = plane % 7 + 1 + var j = 0 + var ddf = 0 + var ddr = 0 + for (df, dr) in encDirs() { + if j == dirTarget { ddf = df; ddr = dr } + j += 1 + } + return PgChessMove(from, (r + ddr * dist) * 8 + (f + ddf * dist), 0) + } + + static fn isKnightMove(df: i32, dr: i32): bool { + let a = Math.abs(df) + let b = Math.abs(dr) + return (a == 1 && b == 2) || (a == 2 && b == 1) + } + + static fn knightIndex(df: i32, dr: i32): i32 { + var i = 0 + var found = -1 + for (kdf, kdr) in encKnights() { + if kdf == df && kdr == dr { found = i } + i += 1 + } + return found + } + + static fn dirIndex(sdf: i32, sdr: i32): i32 { + var i = 0 + var found = -1 + for (ddf, ddr) in encDirs() { + if ddf == sdf && ddr == sdr { found = i } + i += 1 + } + return found + } +} + +// ── Two-headed policy/value net inference (single-source forward pass) ──────────────────────── +// Mirrors Core/Nn/PolicyValueNet: a ReLU trunk → policy logits (one per action, linear) + a scalar +// value (linear). INFERENCE ONLY — training stays on the SDK autograd/GEMM. Weights are float32 in +// the .ckpt; the per-platform parser (chess-net.ts / the C# parity loader) loads them into these f64 +// flat arrays. Trunk weights/biases are concatenated in layer order (each row-major [in, out]) with +// per-layer offsets from `hidden` — a single List avoids nested-generic (List>) params, +// which the transpiler mishandles (same shape as fruitcake's PgDuelingNet). + +// forward() output: raw policy logits (length actions) + the raw scalar value (linear — the leaf +// evaluator applies tanh, matching PolicyValueNet's linear value head + SelfPlayCampaign.Evaluate). +record PgNetOut(logits: List, value: f64) + +class PgPolicyValueNet { + var inputSize: i32 + var actions: i32 + var hidden: List // trunk hidden widths (from the checkpoint) + var trunkWFlat: List // trunk weight matrices concatenated in layer order (each row-major [in, out]) + var trunkBFlat: List // trunk biases concatenated in layer order + var policyW: List // [lastHidden * actions] + var policyB: List // [actions] + var valueW: List // [lastHidden * 1] + var valueB: List // [1] + + init(inputSize: i32, actions: i32, hidden: List, trunkWFlat: List, trunkBFlat: List, policyW: List, policyB: List, valueW: List, valueB: List) { + this.inputSize = inputSize + this.actions = actions + this.hidden = hidden + this.trunkWFlat = trunkWFlat + this.trunkBFlat = trunkBFlat + this.policyW = policyW + this.policyB = policyB + this.valueW = valueW + this.valueB = valueB + } + + // obs (length inputSize) → logits (length actions) + scalar value. Matches PolicyValueNet.Forward on one row. + fn forward(obs: List): PgNetOut { + var x = obs + var prev = this.inputSize + var wOff = 0 + var bOff = 0 + for l in 0..this.hidden.count { + let h = this.hidden[l] + x = relu(linear(x, this.trunkWFlat, wOff, this.trunkBFlat, bOff, prev, h)) + wOff += prev * h + bOff += h + prev = h + } + let logits = linear(x, this.policyW, 0, this.policyB, 0, prev, this.actions) + let value = linear(x, this.valueW, 0, this.valueB, 0, prev, 1) + return PgNetOut(logits, value[0]) + } + + // Dense layer: out[o] = b[bOff+o] + Σ_i x[i]·w[wOff + i*outDim + o] (w row-major [inDim, outDim]). + static fn linear(x: List, w: List, wOff: i32, b: List, bOff: i32, inDim: i32, outDim: i32): List { + var out: List = List() + for o in 0..outDim { + var s = b[bOff + o] + for i in 0..inDim { s += x[i] * w[wOff + i * outDim + o] } + out.add(s) + } + return out + } + + static fn relu(x: List): List { + var out: List = List() + for v in x { out.add(Math.max(0.0, v)) } + return out + } +} + +// ── Inference MCTS (AlphaZero PUCT) ─────────────────────────────────────────────────────────── +// Single-source port of Core/Planning/Mcts, minus the self-play-only Dirichlet root noise (this is +// the browser/serving path). Each simulation walks the tree maximizing Q + U, expands one leaf, and +// backs the net value up the path NEGATING it every ply (zero-sum). `search` returns the root +// visit-count distribution over the 4672 action space; `chooseMove` is its argmax. + +// A leaf evaluation: priors aligned to the node's legal-move list (masked-softmax over legal) + the +// tanh value from the side-to-move's perspective. +record PgLeafEval(priors: List, value: f64) + +class PgMctsNode { + var moves: List // legal action indices (empty if terminal) + var p: List // prior per move (aligned to moves) + var n: List // visit count per move + var w: List // summed value per move (this node's mover perspective) + var children: List + var expanded: bool + var terminal: bool + var terminalValue: f64 // set when terminal: -1 loss (side-to-move mated) / 0 draw, mover-relative + + init(moves: List, terminal: bool, terminalValue: f64) { + this.moves = moves + this.terminal = terminal + this.terminalValue = terminalValue + this.p = List() + this.n = List() + this.w = List() + this.children = List() + this.expanded = false + } +} + +class PgChessMcts { + // Run `sims` PUCT simulations from `root` and return the root visit-count distribution over the + // 4672 action space (0 for illegal/unvisited). Falls back to raw priors if nothing was visited. + static fn search(net: PgPolicyValueNet, root: PgChessState, sims: i32, cpuct: f64): List { + let rootNode = newNode(root) + if !rootNode.terminal { + expandLeaf(rootNode, net, root) + for s in 0..sims { simulate(rootNode, net, root, cpuct) } + } + + var pi: List = List() + for i in 0..4672 { pi.add(0.0) } + var total = 0 + for i in 0..rootNode.moves.count { total += rootNode.n[i] } + if total == 0 { + for i in 0..rootNode.moves.count { pi[rootNode.moves[i]] = rootNode.p[i] } + return pi + } + for i in 0..rootNode.moves.count { pi[rootNode.moves[i]] = rootNode.n[i] * 1.0 / total } + return pi + } + + // Best action index (argmax visit count via the returned distribution). + static fn chooseMove(net: PgPolicyValueNet, root: PgChessState, sims: i32, cpuct: f64): i32 { + let pi = search(net, root, sims, cpuct) + var best = -1 + var bestV = -1.0 + for i in 0..pi.count { if pi[i] > bestV { bestV = pi[i]; best = i } } + return best + } + + // Returns the value of `state` from the perspective of its side to move. + static fn simulate(node: PgMctsNode, net: PgPolicyValueNet, state: PgChessState, cpuct: f64): f64 { + if node.terminal { return node.terminalValue } + if !node.expanded { return expandLeaf(node, net, state) } + + let edge = selectChild(node, cpuct) + let childState = state.applyIndex(node.moves[edge]) + if node.children[edge] == null { node.children[edge] = newNode(childState) } + let child = node.children[edge]! + let value = -simulate(child, net, childState, cpuct) // flip: child mover's value + node.n[edge] = node.n[edge] + 1 + node.w[edge] = node.w[edge] + value + return value + } + + static fn selectChild(node: PgMctsNode, cpuct: f64): i32 { + var sumN = 0 + for i in 0..node.n.count { sumN += node.n[i] } + let sqrtSum = Math.sqrt(sumN * 1.0) + var best = 0 + var bestScore = -1.0e30 + for i in 0..node.moves.count { + let q = if node.n[i] > 0 { node.w[i] / node.n[i] } else { 0.0 } + let u = cpuct * node.p[i] * sqrtSum / (1.0 + node.n[i]) + let score = q + u + if score > bestScore { bestScore = score; best = i } + } + return best + } + + static fn newNode(state: PgChessState): PgMctsNode { + let r = state.result() // 0 ongoing, 1 loss (side-to-move mated), 2 draw + if r != 0 { + let tv = if r == 1 { -1.0 } else { 0.0 } + return PgMctsNode(List(), true, tv) + } + return PgMctsNode(state.legalMoveIndices(), false, 0.0) + } + + // Evaluate the leaf, seed per-move priors (masked-softmax over legal), return the net's value estimate. + static fn expandLeaf(node: PgMctsNode, net: PgPolicyValueNet, state: PgChessState): f64 { + let ev = evaluate(net, state, node.moves) + let k = node.moves.count + node.p = List() + node.n = List() + node.w = List() + node.children = List() + for i in 0..k { + node.p.add(ev.priors[i]) + node.n.add(0) + node.w.add(0.0) + node.children.add(null) + } + node.expanded = true + return ev.value + } + + // Masked-softmax priors over `moves` (aligned) + tanh value. Mirrors SelfPlayCampaign.Evaluate. + static fn evaluate(net: PgPolicyValueNet, state: PgChessState, moves: List): PgLeafEval { + let out = net.forward(state.writeObservation()) + var mx = -1.0e30 + for m in moves { if out.logits[m] > mx { mx = out.logits[m] } } + var pr: List = List() + var sum = 0.0 + for m in moves { + let e = Math.exp(out.logits[m] - mx) + pr.add(e) + sum += e + } + if sum > 0.0 { + for i in 0..pr.count { pr[i] = pr[i] / sum } + } else { + for i in 0..pr.count { pr[i] = 1.0 / pr.count } + } + return PgLeafEval(pr, Math.tanh(out.value)) + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs new file mode 100644 index 0000000..162f5cc --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs @@ -0,0 +1,100 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +/// A Connect-4 position: 7×6 cells (0 = empty, 1 / 2 = the two players) plus the side to move. +/// Immutable from the caller's view — always returns a fresh state. +public sealed class Connect4State +{ + public const int Columns = 7; + public const int Rows = 6; + + /// Row-major, row 0 = bottom: cell (row, col) = Cells[row * Columns + col]. + public byte[] Cells { get; } + public int ToMove { get; } // 1 or 2 + + public Connect4State(byte[] cells, int toMove) { Cells = cells; ToMove = toMove; } +} + +/// +/// Connect-4 as an — the cheap first consumer of the self-play stack (tiny rules, +/// converges from random self-play in minutes on CPU, and a shallow negamax gives an exact test oracle). The action +/// index is the column (0..6); the observation is a side-to-move-relative two-plane (mine / theirs) board. +/// +public sealed class Connect4Game : IZeroSumGame +{ + public int PolicySize => Connect4State.Columns; // one action per column + public int ObservationSize => 2 * Connect4State.Columns * Connect4State.Rows; // mine + theirs planes + + public Connect4State Root(ulong? seed = null) => new(new byte[Connect4State.Columns * Connect4State.Rows], toMove: 1); + + public IReadOnlyList LegalMoves(Connect4State state) + { + var moves = new List(Connect4State.Columns); + int topRowBase = (Connect4State.Rows - 1) * Connect4State.Columns; + for (int col = 0; col < Connect4State.Columns; col++) + if (state.Cells[topRowBase + col] == 0) moves.Add(col); // top cell empty → column not full + return moves; + } + + public Connect4State Apply(Connect4State state, int move) + { + var cells = (byte[])state.Cells.Clone(); + for (int row = 0; row < Connect4State.Rows; row++) + { + int idx = row * Connect4State.Columns + move; + if (cells[idx] == 0) { cells[idx] = (byte)state.ToMove; break; } + } + return new Connect4State(cells, toMove: 3 - state.ToMove); + } + + public GameResult Result(Connect4State state) + { + int opponent = 3 - state.ToMove; + if (HasFour(state.Cells, opponent)) return GameResult.Loss; // the player who just moved completed a line + if (HasFour(state.Cells, state.ToMove)) return GameResult.Win; // defensive; unreachable in normal play + return IsFull(state.Cells) ? GameResult.Draw : GameResult.Ongoing; + } + + public void WriteObservation(Connect4State state, Span destination) + { + int n = Connect4State.Columns * Connect4State.Rows; + int mine = state.ToMove, theirs = 3 - state.ToMove; + for (int i = 0; i < n; i++) + { + destination[i] = state.Cells[i] == mine ? 1f : 0f; + destination[n + i] = state.Cells[i] == theirs ? 1f : 0f; + } + } + + private static bool IsFull(byte[] cells) + { + int topRowBase = (Connect4State.Rows - 1) * Connect4State.Columns; + for (int col = 0; col < Connect4State.Columns; col++) + if (cells[topRowBase + col] == 0) return false; + return true; + } + + // True if player p has any four-in-a-row (horizontal, vertical, or either diagonal). + private static bool HasFour(byte[] cells, int p) + { + const int C = Connect4State.Columns, R = Connect4State.Rows; + for (int row = 0; row < R; row++) + for (int col = 0; col < C; col++) + { + if (cells[row * C + col] != p) continue; + if (col + 3 < C && Line(cells, p, row, col, 0, 1)) return true; // → + if (row + 3 < R && Line(cells, p, row, col, 1, 0)) return true; // ↑ + if (row + 3 < R && col + 3 < C && Line(cells, p, row, col, 1, 1)) return true; // ↗ + if (row + 3 < R && col - 3 >= 0 && Line(cells, p, row, col, 1, -1)) return true; // ↖ + } + return false; + } + + private static bool Line(byte[] cells, int p, int row, int col, int dRow, int dCol) + { + for (int step = 1; step < 4; step++) + if (cells[(row + step * dRow) * Connect4State.Columns + (col + step * dCol)] != p) return false; + return true; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs new file mode 100644 index 0000000..e0fade2 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs @@ -0,0 +1,48 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +/// +/// A depth-limited negamax solver for Connect-4 — the exact oracle the self-play tests measure MCTS against (the same +/// role Kociemba plays for the cube and BFS for Rush Hour). Scores are from the side-to-move's perspective: +/// +1 = a forced win within the horizon, -1 = a forced loss, 0 = a draw or undecided within the +/// depth. Full-game solving is exponential, so callers pass a modest maxDepth and use it on tactical positions. +/// +public static class Connect4Solver +{ + private static readonly Connect4Game Game = new(); + + /// The best move for the side to move (preferring a forced win, else a draw, avoiding a loss) and its + /// negamax score in {-1, 0, +1} within plies. + public static (int Score, int BestMove) Solve(Connect4State state, int maxDepth) + { + int bestScore = int.MinValue, bestMove = -1; + foreach (int move in Game.LegalMoves(state)) + { + int score = -Negamax(Game.Apply(state, move), maxDepth - 1); + if (score > bestScore) { bestScore = score; bestMove = move; } + } + return (bestScore == int.MinValue ? 0 : bestScore, bestMove); + } + + /// The forced result of for its side to move within plies. + public static int Negamax(Connect4State state, int depth) + { + switch (Game.Result(state)) + { + case GameResult.Loss: return -1; // side to move has already lost + case GameResult.Win: return 1; + case GameResult.Draw: return 0; + } + if (depth <= 0) return 0; // undecided within the horizon → treat as neutral + + int best = -2; // below the -1..+1 range + foreach (int move in Game.LegalMoves(state)) + { + int value = -Negamax(Game.Apply(state, move), depth - 1); + if (value > best) best = value; + if (best == 1) break; // a forced win can't be improved on — prune + } + return best; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index c81fcff..435a3d1 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -4,12 +4,14 @@ - + - +