From 78899b9ff4ff0b36b320d15bc76d433d79c803ca Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 21:19:24 +0200 Subject: [PATCH 01/33] =?UTF-8?q?docs(M39):=20PRD=20+=20plan=20for=20self-?= =?UTF-8?q?play=20training=20(Connect-4=20=E2=86=92=20chess)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A three-agent investigation established that ~70% of the outer loop already exists (PolicyValueNet, the soft-CE+value train step, ITrainingCampaign/ CampaignRunner, model store + checkpoint format, the M38 AdamState/TrainWindow/ PolicyGrowth plumbing, action masking, RNG streams, --viz, the A/B harness). Adds docs/prd/CHESS_SELFPLAY_PRD.md and an M39 milestone in PLAN.md: AlphaZero-style self-play (MCTS + the two-headed net) on ONE new deep seam, IZeroSumGame in Core/Planning (a sibling to IDeterministicModel). Phased to de-risk the novel machinery on Connect-4 first (cheap CPU convergence + a negamax oracle), then land chess as the seam's second consumer (perft-gated legal movegen). Honest scope: MLP-only net → legal, steadily- improving play, not engine strength; self-play is CPU-bound. Docs only. Stacked on the M38 branch (reuses its Lab plumbing). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/CHESS_SELFPLAY_PRD.md | 113 +++++++++++++++++++++++++++++++++ docs/prd/PLAN.md | 40 ++++++++++++ 2 files changed, 153 insertions(+) create mode 100644 docs/prd/CHESS_SELFPLAY_PRD.md diff --git a/docs/prd/CHESS_SELFPLAY_PRD.md b/docs/prd/CHESS_SELFPLAY_PRD.md new file mode 100644 index 0000000..9df073b --- /dev/null +++ b/docs/prd/CHESS_SELFPLAY_PRD.md @@ -0,0 +1,113 @@ +# Self-play training (chess) — PRD + +**Status:** Planned · 2026-07-12 · branch TBD (off `master`) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M39 · **Depends on:** the Core NN + checkpoint layer (§2/§5 of [../ARCHITECTURE.md](../ARCHITECTURE.md)), the two-headed `PolicyValueNet` (M37), the training-campaign harness (§8), and action masking (§3). Additive — no change to existing training behaviour. + +## 1. Problem + +The SDK trains agents in three paradigms today — model-free RL (DQN/PPO), planning over a learned value net (DAVI), and **imitation from an oracle** (the cube's Kociemba teacher, Rush Hour's BFS teacher). All three need an external signal: a reward function, a forward model, or an *exact oracle*. For a game like **chess** there is no cheap exact oracle, and a reactive policy plateaus — this repo's own history is unambiguous that **search is the lever** (Snake/FruitCake/Cube all show a capped reactive net amplified massively by inference search). + +The missing paradigm is **self-play**: two players (the same improving network) play each other, and the games themselves become the training signal — the network bootstraps from nothing to strong play with no human data and no oracle. That is how a from-scratch SDK trains chess. We want to add self-play as a **first-class, reusable capability** — chess is the headline, but the machinery (a two-player game abstraction + MCTS + a self-play campaign) should be shared so the *next* two-player game plugs in by writing only its rules. + +**Constraint (load-bearing):** reuse the SDK; write as little new code as possible. A three-agent investigation (2026-07-12) established that **~70% of the outer loop already exists** and the genuinely-new code is small and well-isolated (see §5). + +## 2. Goal & success criteria + +- **Self-play works, verifiably (the priority).** Starting from a **randomly-initialized** net, self-play training produces a net whose strength **rises monotonically** against a fixed baseline — measured first on **Connect-4** (converges in minutes on CPU; a negamax oracle gives an exact yardstick), then on **chess** (win-rate vs a random-legal baseline climbs; later, Elo vs a frozen earlier checkpoint). +- **Reuse-first.** The self-play stack reuses `PolicyValueNet` (unchanged), the soft-CE+value training step (already used by the imitation campaigns), `ITrainingCampaign`/`CampaignRunner`, the model store + checkpoint format, the M38 Lab plumbing (`AdamState`/`TrainWindow`/`PolicyGrowth`), action masking, the RNG streams, `--viz`, and the paired-seed A/B harness (→ the Elo promotion gate). New code is confined to: one game seam, MCTS, the self-play data loop, and each game's rules. +- **One new deep seam, not a shallow one.** `IZeroSumGame` in `Core/Planning` is the single new abstraction; it must pass the M38 bar (minimal, honest, consumed by *both* MCTS and the self-play trainer; Connect-4/chess/checkers/Go plug in unchanged). It is a **sibling** to `IDeterministicModel`, not an extension of it (see §3). +- **Correctness gated.** Chess legal-move generation is verified by **perft** (leaf-node counts matched to published values, depths 5–6, from startpos + Kiwipete + standard positions) *before any training*. Nothing downstream is trusted until perft passes. +- **Determinism & resume.** Self-play RNG (Dirichlet noise, move sampling) uses its own `SeedSequence` stream; a campaign resumes bitwise-consistently through the model store, like every other campaign (a `CampaignContractTests` roundtrip proves it). + +**Honest non-goals for v1 (strength).** Not superhuman, not engine-strength chess. The net is **MLP-only** (no conv/attention in the backend), so a flattened board loses spatial structure — this proves the pipeline and gives *steadily-improving, fully-legal* play, but caps positional strength. And a small chess MLP does **not** clear the GPU routing threshold that helps the cube, so self-play is CPU-bound. The realistic target is *"plays fully-legal, steadily-improving chess"* — Connect-4 is where the self-improvement curve is unmistakable; chess is the headline consumer of the same rails. Conv-backend support and true engine strength are explicitly out of scope (separate workstreams). + +**Other non-goals (v1).** No DQN/PPO for self-play (wrong fit — a ~4600-wide masked Q-head over terminal-only sparse reward under self-play non-stationarity); no reuse of the DQN `ReplayBuffer` (it's `(s,a,r,s′,term,mask)`-shaped; self-play wants an `(obs,π,z)` window — a plain list, like the imitation campaigns); no bitboards unless a bench says movegen is the bottleneck; no web showcase page in phase 1. + +## 3. Key decision — AlphaZero-style, on one new game seam, Connect-4 first + +**Design it twice** (full analysis in the investigation): + +- **Design A — plain self-play over `IEnvironment`.** Model the game side-to-move, reward = terminal outcome, opponent = the current/frozen net (a small `IOpponentPolicy` seam), train with PPO/REINFORCE. *Rejected as the endpoint:* no lookahead → tactically-blind play, the exact reactive plateau the repo documents. Useful only as a plumbing pipe-test. +- **Design B — AlphaZero-style (chosen).** MCTS-guided self-play: each move runs PUCT simulations whose leaves are scored by the two-headed net; the move is sampled from the **root visit-count distribution** π; training targets are `(π, z)` where `z` is the game outcome from that position's mover. Loss = `CE(π, policyLogits) + MSE(z, tanh(value))`. + +**Why B, given "minimal new code":** the chess rules engine is the *same irreducible cost* in both designs. B's only marginal cost over A is `Mcts.cs` + visit-count targets instead of an opponent seam + PPO glue — and B is the design that makes "two players play each other and the model improves" actually *true* for a tactically deep game. It also showcases the SDK's signature composition (game model + net + search), which is the repo's whole thesis. + +**Why a new seam, not `IEnvironment` or `IDeterministicModel`:** `IEnvironment` is single-agent (single scalar reward, episode loop) — it can host Design A but not MCTS. `IDeterministicModel` is *close* — its non-mutating `Apply` is exactly right — but its `IsGoal` (single bool) can't express **win/loss/draw relative to the side to move**, its `ActionCount` (valid in every state) lies for chess's per-state legal moves, and it carries no observation/policy-size. Bolting those on would leak (a shallow, dishonest interface). So add a minimal sibling: + +```csharp +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// Terminal result RELATIVE TO THE SIDE TO MOVE in the queried state. +public enum GameResult { Ongoing, Win, Loss, Draw } + +/// +/// A perfect-information, two-player, zero-sum, deterministic game — the shared shape MCTS and a self-play +/// trainer consume. Distinct from IEnvironment (no reward/episode loop) and IDeterministicModel (side-to-move +/// + win/loss/draw + per-state legal moves, not a single goal). Apply returns a FRESH successor and leaves +/// 'state' untouched (search reuses a state across moves); after Apply the side to move has flipped. +/// +public interface IZeroSumGame +{ + int PolicySize { get; } // fixed action-index space = the net's policy-head width + int ObservationSize { get; } + TState Root(ulong? seed = null); + IReadOnlyList LegalMoves(TState state); + TState Apply(TState state, int move); // non-mutating; flips side to move + GameResult Result(TState state); // terminal result for the mover, or Ongoing + void WriteObservation(TState state, Span destination); // side-to-move-relative encoding +} +``` + +**Why Connect-4 first:** it de-risks the *novel* machinery (MCTS soundness, value-sign backup, self-play target generation, non-collapse) on a game with trivial, fast, easily-correct rules and a cheap **negamax** oracle for an exact test — *separately* from chess's large, silent-bug-prone rules surface. Chess then lands as the seam's **second consumer**, riding the same MCTS/campaign/net/checkpoint rails unchanged. This is the same "prove the mechanism cheaply, then scale" discipline the rest of the repo follows, and each piece stays independently verifiable. + +## 4. Design — the pieces + +| Piece | Where | New/Reused | +|---|---|---| +| `IZeroSumGame` + `GameResult` | `Core/Planning/IZeroSumGame.cs` | **new** (deep seam) | +| `Mcts` — PUCT search: select → expand-with-priors → sign-flipping value backup; Dirichlet root noise; temperature; returns the root visit-count π | `Core/Planning/Mcts.cs` | **new** (~300–450 LOC; the core novel algorithm — no tree search exists in the repo) | +| `PolicyValueTraining.TrainStep(net, adam, obs[], policyTargets[], valueTargets[], batch)` — soft-CE + value regression, grad-clip, Adam | `Core/Training/` (generalized from `CubePolicyTraining.TrainStep`) | **new-ish** (~60 LOC; cube-imitation can also call it) | +| `SelfPlayCampaign : ITrainingCampaign, INetworkTelemetrySource` — `TrainChunk` plays K self-play games (MCTS both sides via net leaf) → `(obs,π,z)` rolling window → train → `PolicyGrowth.Maybe`; `Evaluate` = arena win-rate vs frozen best (→ Elo); `Checkpoint` = save-best on win-rate + `AdamState.Save` | `tools/…Lab/` | **new** (mirrors `CubeImitationCampaign`; reuses `AdamState`/`TrainWindow`/`PolicyGrowth`/telemetry) | +| `PolicyValueNet` — the AlphaZero net, **unchanged**; value wrapped in `tanh` at the call site for WDL in [-1,1] | `Core/Nn/PolicyValueNet.cs` | **reused** | +| `Connect4Game : IZeroSumGame<…>` + a negamax oracle (test) | `Environments/Connect4/` | **new** (small) | +| `ChessBoard` + legal movegen + draw/mate detection; move encoding (8×8×73 = 4672); `ChessGame : IZeroSumGame` / `ChessEnv`; plane observation; `ChessPolicyNet` wrapper; perft tests | `Environments/Chess/` | **new** (the large, test-heavy chunk) | +| Elo eval (`…Ab`), `--game connect4|chess` Lab dispatch, checkpoints via the model store + Git LFS | reuse patterns | **reused** | + +**MCTS composition** mirrors `ValueGuidedSearch` over `IDeterministicModel`: + +```csharp +public static class Mcts +{ + public sealed record Config(int Simulations = 400, float Cpuct = 1.25f, + float DirichletAlpha = 0.3f, float RootNoiseFrac = 0.25f); + + /// Leaf evaluator: priors over PolicySize + value in [-1,1] for the mover (PolicyValueNet under NoGrad). + public delegate (float[] Priors, float Value) Evaluate(TState state); + + /// Search from 'state'; return the root visit-count distribution over PolicySize (the training target π). + public static float[] Search(IZeroSumGame game, TState state, + Evaluate evaluate, Config config, Xoshiro256StarStar rng); +} +``` + +## 5. Reuse-vs-new ledger (rough) + +**Reused verbatim (0 new lines):** `PolicyValueNet` + Adam + tape autograd + `LogSoftmax`/`Mul`/`Sum`/`Tanh`; `ITrainingCampaign`/`CampaignRunner`/`AIHost`; `IModelStore`/`FileModelStore` + `CheckpointFormat` + `PolicyValueNet.Save/Load` (generic over a `kind` string, v2 trunk-widths); `IEnvironment`/`Space`/`IActionMaskProvider`/`IStatefulEnvironment`; `Net2Net` growth; `--viz`; the A/B harness pattern; `SeedSequence` streams. + +**New:** the game seam + MCTS + the `PolicyValueTraining` step + the self-play campaign (~900–1100 LOC total, most of it MCTS + the campaign) — plus each game's rules: Connect-4 (small), **chess (~1,500–2,200 LOC, rules + perft dominate)**. Total ≈ 3,000–4,500 LOC, of which the chess rules engine is the bulk and MCTS is the subtle part. + +## 6. Risks + +1. **Chess-rule correctness (highest).** Castling (through/into check), en passant (incl. discovered check), promotion, pins, and draw rules (50-move, threefold via position hashing, insufficient material, stalemate vs checkmate) are silent-bug territory. **Mitigation, non-optional:** perft node-count tests to depth 5–6 from standard positions matched exactly to published counts, *before* any training. Isolating this in M39.2 (chess) — after the rails are proven on Connect-4 in M39.1 — means a movegen bug can't masquerade as an MCTS/self-play bug. +2. **MCTS soundness / self-play collapse.** Wrong value-sign backup, missing Dirichlet root noise, or no temperature schedule → collapsing targets. Mitigation: unit-test MCTS on forced-win/forced-draw Connect-4 positions against negamax; gate training on rising win-rate vs a *frozen* baseline. +3. **Compute.** Self-play games are long; MCTS is hundreds of sims/move. CPU-bound (the small MLP won't clear the GPU threshold). Mitigation: Connect-4 for the convergence proof; keep chess sims modest; batched-leaf MCTS is a phase-3 lever. Set expectations: legal, improving play — not engine strength. +4. **Representation ceiling.** MLP over flattened planes has no spatial prior. Accepted for v1; conv backend is a separate workstream if positional strength becomes a goal. + +## 7. Verification + +- **M39.1 gate (Connect-4):** perft-free (tiny rules, unit-tested directly); MCTS unit tests vs negamax on forced positions; from random init, self-play win-rate vs negamax/random **climbs**; `CampaignContractTests` resume roundtrip (fresh → `TrainChunk` advances → `Checkpoint` → new instance `Resume`s and continues); `dotnet test --filter "Category!=Slow"` green. +- **M39.2 gate (chess):** **perft matches published counts** (startpos, Kiwipete, …) to depth 5–6 — the hard gate; move-encode/decode round-trips over all legal moves; env terminated-vs-truncated split correct; win-rate vs random-legal climbs; contract-test resume; ship `models/chess.az.ckpt` (LFS). +- Every step ends on a green build + its gate, revert-friendly, one milestone at a time. + +See [PLAN.md](PLAN.md) M39 for the phased step order. diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index d0bb485..9bd36de 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1359,6 +1359,46 @@ shallow/leaky, PR #27's correct call), the 3 imitation/policy campaigns *as whol policy campaign SHA256-bitwise-identical vs baseline; `--viz` live for every game; net **negative** line diff with every shared module carrying an interface comment. See `BOILERPLATE_REDUCTION_PRD.md`. +## M39 — Self-play training (Connect-4 → chess) *(planned 2026-07-12; see `CHESS_SELFPLAY_PRD.md`)* 🔜 + +**Problem.** The SDK trains via reward (DQN/PPO), a forward model (DAVI), or an exact oracle (cube/Rush Hour imitation). Chess +has no cheap oracle and a reactive net plateaus — the repo's own history says **search is the lever**. The missing paradigm is +**self-play**: the improving net plays itself and the games *are* the training signal (AlphaZero-style). Add it as a **reusable** +capability — chess is the headline, but the machinery is shared so the next two-player game plugs in by writing only its rules. + +**Key decision (design-it-twice, 3-agent investigation 2026-07-12).** AlphaZero-style (MCTS-guided self-play + the two-headed +`PolicyValueNet`), **not** plain reactive self-play (tactically blind — the documented plateau). The chess rules engine is the +same irreducible cost either way, so the marginal cost of "real improvement" is just MCTS + visit-count targets. One new **deep** +seam — `IZeroSumGame` in `Core/Planning` (a *sibling* to `IDeterministicModel`: side-to-move + win/loss/draw + per-state +legal moves, which `IDeterministicModel` can't honestly express) — consumed by both MCTS and the self-play campaign. **Reuse-first:** +~70% of the outer loop already exists (`PolicyValueNet` unchanged, the soft-CE+value train step from the imitation campaigns, +`ITrainingCampaign`/`CampaignRunner`, model store + checkpoint format, the M38 `AdamState`/`TrainWindow`/`PolicyGrowth` plumbing, +action masking, RNG streams, `--viz`, the A/B harness → Elo gate). New code is confined to the seam, MCTS, the `(obs,π,z)` +self-play loop, and each game's rules. + +**Phased plan (each step ends on a green build + its gate; ordered to de-risk the *novel* machinery before the huge rules surface).** + +- **M39.1 — the rails, proven on Connect-4.** `IZeroSumGame` + `Core/Planning/Mcts.cs` (PUCT, sign-flipping value backup, + Dirichlet root noise, temperature → root visit-count π) + `PolicyValueTraining.TrainStep` (generalized from + `CubePolicyTraining.TrainStep`: soft-CE + value regression) + `SelfPlayCampaign : ITrainingCampaign` (reusing + `AdamState`/`TrainWindow`/`PolicyGrowth`/telemetry) + `Connect4Game : IZeroSumGame` + a **negamax** oracle for tests. **Gate:** + MCTS unit-tested vs negamax on forced-win/draw positions; from random init, self-play win-rate vs negamax/random **climbs**; + `CampaignContractTests` resume roundtrip; fast suite green. This validates the self-play machinery cheaply, away from chess rules. +- **M39.2 — chess as consumer #2.** `ChessBoard` + full legal movegen + draw/mate detection; the 8×8×73 = 4672 move encoding; + `ChessGame`/`ChessEnv` (`IEnvironment` + `IActionMaskProvider` + `IStatefulEnvironment`); flattened-plane observation (static, + train==serve); a thin `ChessPolicyNet` wrapper over `PolicyValueNet` (policy head 4672, value `tanh` → WDL); `ChessSelfPlayCampaign` + + `--game chess` Lab dispatch; an Elo eval (`ChessAb`, mirroring `FruitCakeAb`). **Gate (the hard one): perft** node-counts matched + to published values (startpos, Kiwipete, …) to depth 5–6 *before any training*; move encode/decode round-trips; terminated-vs-truncated + split correct; win-rate vs random-legal climbs; contract-test resume; ship `models/chess.az.ckpt` (LFS). +- **M39.3 — scale (optional).** Batched-leaf MCTS; GPU-resident forward *if* the net grows enough to clear the routing threshold + (honest: a small chess MLP won't); conv-backend support for positional strength (a separate backend workstream); a web showcase page. + +**Honest scope.** MLP-only net (no conv) over a flattened board → *legal, steadily-improving* play, not engine strength; self-play is +CPU-bound (the small MLP won't hit the GPU lever that helps the cube). Connect-4 is where the self-improvement curve is unmistakable; +chess is the headline consumer. **Left out of v1:** DQN/PPO for self-play (wrong fit), the DQN `ReplayBuffer` (uses an `(obs,π,z)` +window instead), bitboards (unless a bench forces it), superhuman strength. **Whole-milestone gate:** builds 0/0; fast suite green; +perft passes; both games' self-play win-rate rises vs a fixed baseline; resume roundtrips. See `CHESS_SELFPLAY_PRD.md`. + ## Testing strategy (cross-cutting, from research) 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket. From e017afa62c02bd714a814af93d348d8cb1d4b948 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 21:43:22 +0200 Subject: [PATCH 02/33] M39.1: self-play rails (IZeroSumGame + MCTS + SelfPlayCampaign) on Connect-4 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reusable self-play stack, proven on Connect-4 before chess's rules surface (PLAN M39.1). New and self-contained: - Core/Planning/IZeroSumGame — the two-player zero-sum game seam (side-to-move + win/loss/draw + per-state legal moves), a sibling to IDeterministicModel that MCTS and the self-play trainer both consume. - Core/Planning/Mcts.cs — AlphaZero PUCT: select (Q+U) → expand-with-priors → net-leaf eval → value backup negated every ply (zero-sum), Dirichlet root noise, returns the root visit-count π. Composes over the seam + an Evaluate delegate exactly as ValueGuidedSearch composes over IDeterministicModel. - Environments/Connect4/{Connect4Game : IZeroSumGame, Connect4Solver} — the cheap first consumer + a depth-limited negamax test oracle. - Lab/PolicyValueTraining.TrainStep — soft-CE(π) + MSE(tanh value, z), the AlphaZero loss (generalized from CubePolicyTraining to raw arrays). - Lab/SelfPlayCampaign : ITrainingCampaign — plays K MCTS self-play games/chunk → (obs, π, z) rolling window → trains PolicyValueNet; Evaluate = win-rate vs a random-legal opponent; reuses AdamState + TrainWindow + telemetry + the model store/checkpoint format. --game connect4 dispatch. REUSED unchanged: PolicyValueNet (the AZ net), Adam, the autograd ops, ITrainingCampaign/CampaignRunner/AIHost, IModelStore + checkpoint format, AdamState/TrainWindow, SeedSequence streams, --viz telemetry. (Growth via PolicyGrowth deferred — it needs an IGrowableTrunkNet wrapper.) Gate met: 5 fast Connect-4/MCTS tests green (incl. MCTS finding a forced win purely by search, agreeing with negamax) + the Slow self-play resume-roundtrip contract; 331 fast tests total green. A 4-minute `--game connect4` run from random init reaches 100% vs a random opponent with a falling policy loss (1.37→1.22) — the pipeline learns end-to-end. (A discriminating frozen-baseline arena lands with chess in M39.2.) Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Planning/IZeroSumGame.cs | 56 ++++ .../Planning/Mcts.cs | 178 +++++++++++ .../Connect4/Connect4Game.cs | 100 +++++++ .../Connect4/Connect4Solver.cs | 48 +++ .../Connect4Tests.cs | 93 ++++++ .../SelfPlayCampaignTests.cs | 57 ++++ .../Connect4Lab.cs | 31 ++ .../PolicyValueTraining.cs | 34 +++ .../Program.cs | 1 + .../SelfPlayCampaign.cs | 277 ++++++++++++++++++ 10 files changed, 875 insertions(+) create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs create mode 100644 tests/MintPlayer.AI.ReinforcementLearning.Tests/Connect4Tests.cs create mode 100644 tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs create mode 100644 tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs create mode 100644 tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyValueTraining.cs create mode 100644 tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs new file mode 100644 index 0000000..c6aabc0 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/IZeroSumGame.cs @@ -0,0 +1,56 @@ +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// A terminal game result, stated RELATIVE TO THE SIDE TO MOVE in the queried state. +public enum GameResult +{ + /// Not a terminal position. + Ongoing, + /// The side to move has already won (rare — most games are won on the mover's own move). + Win, + /// The side to move has lost (e.g. it is checkmated / the opponent just completed a line). + Loss, + /// A draw (stalemate, repetition, full board, …). + Draw, +} + +/// +/// A perfect-information, two-player, zero-sum, deterministic game — the shared shape that MCTS +/// () and a self-play trainer both consume. It is deliberately distinct from +/// IEnvironment (which has a single-agent reward/episode loop) and from +/// (whose single-goal IsGoal and always-valid ActionCount can't express side-to-move, +/// win/loss/draw, or per-state legal moves). +/// +/// Conventions the implementer MUST honour: returns a FRESH successor and leaves the input +/// state untouched (search reuses one state across many candidate moves), and applying a move FLIPS the side +/// to move. and are always from the perspective of the side to +/// move in the given state, so the network sees one canonical "me vs. them" view regardless of whose turn it is. +/// +/// +/// The immutable-from-the-caller's-view game state (a position). +public interface IZeroSumGame +{ + /// The fixed action-index space — equals the policy head width of the net that plays this game. + /// A given state's legal subset is ; every legal move is an index in [0, PolicySize). + int PolicySize { get; } + + /// The length of the observation writes (the net's input width). + int ObservationSize { get; } + + /// The start position. is for games with a randomized setup (chess/Connect-4 ignore it). + TState Root(ulong? seed = null); + + /// The legal action indices in (a subset of [0, PolicySize)). Never empty for a + /// non-terminal state. + IReadOnlyList LegalMoves(TState state); + + /// The successor after playing — a fresh state, with the side to move flipped. + /// is not mutated. must be one of . + TState Apply(TState state, int move); + + /// The terminal result for the side to move in , or . + GameResult Result(TState state); + + /// Writes the side-to-move-relative observation of into + /// (length ). + void WriteObservation(TState state, Span destination); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs new file mode 100644 index 0000000..f85f39f --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Planning/Mcts.cs @@ -0,0 +1,178 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Planning; + +/// +/// AlphaZero-style PUCT Monte-Carlo Tree Search over an , guided by a neural +/// leaf evaluator (policy priors + a value in [-1,1]). It composes over the game seam + an +/// delegate exactly as composes over + a +/// cost-to-go delegate, so Core carries no game or NN dependency. +/// +/// Each simulation walks the tree by maximizing Q + U (mean action value + a prior-weighted exploration +/// bonus), expands one new leaf, evaluates it with the net, and backs the value up the path negating it every +/// ply (zero-sum: a position good for the child's mover is bad for the parent's). At the root, Dirichlet noise is +/// mixed into the priors so self-play keeps exploring. returns the root's visit-count +/// distribution — the policy target π for training, from which the caller samples (early game) or takes the argmax. +/// +/// +public static class Mcts +{ + /// Tree simulations per move (more = stronger, slower). + /// PUCT exploration constant. + /// Concentration of the root-noise Dirichlet (chess ≈ 0.3; smaller = spikier). + /// Fraction of the root prior replaced by Dirichlet noise (0 disables). + public sealed record Config(int Simulations = 200, float Cpuct = 1.25f, float DirichletAlpha = 0.3f, float RootNoiseFrac = 0.25f); + + /// Evaluates a leaf: a probability distribution over PolicySize (illegal moves ≈ 0; the search + /// renormalizes over the legal set) and the position value in [-1,1] from the side-to-move's perspective. Backed + /// by a PolicyValueNet.Forward under NoGrad, softmaxed over legal moves + tanh value. + public delegate (float[] Priors, float Value) Evaluate(TState state); + + private sealed class Node + { + public required int[] Moves; // legal action indices in this state + public float[] P = []; // prior per move (aligned to Moves) + public int[] N = []; // visit count per move + public float[] W = []; // summed value per move, from THIS node's mover perspective + public Node?[] Children = []; + public bool Expanded; + public bool Terminal; + public float TerminalValue; // set when Terminal: +1 win / -1 loss / 0 draw, mover-relative + } + + /// Runs .Simulations from and returns the root + /// visit-count distribution over game.PolicySize (sums to 1 over the legal moves; 0 elsewhere). Falls back + /// to the raw priors only if no simulation recorded a visit (Simulations == 0). + public static float[] Search(IZeroSumGame game, TState rootState, + Evaluate evaluate, Config config, Xoshiro256StarStar rng) + { + var root = NewNode(game, rootState); + if (!root.Terminal) + { + ExpandLeaf(root, evaluate, rootState); + AddRootNoise(root, config, rng); + for (int sim = 0; sim < config.Simulations; sim++) + Simulate(root, game, rootState, evaluate, config); + } + + var pi = new float[game.PolicySize]; + int total = 0; + for (int i = 0; i < root.Moves.Length; i++) total += root.N[i]; + if (total == 0) + { + for (int i = 0; i < root.Moves.Length; i++) pi[root.Moves[i]] = root.P[i]; + return pi; + } + for (int i = 0; i < root.Moves.Length; i++) pi[root.Moves[i]] = root.N[i] / (float)total; + return pi; + } + + // Returns the value of 'state' from the perspective of its side to move. + private static float Simulate(Node node, IZeroSumGame game, TState state, + Evaluate evaluate, Config config) + { + if (node.Terminal) return node.TerminalValue; + if (!node.Expanded) return ExpandLeaf(node, evaluate, state); + + int edge = SelectChild(node, config); + var childState = game.Apply(state, node.Moves[edge]); + node.Children[edge] ??= NewNode(game, childState); + float value = -Simulate(node.Children[edge]!, game, childState, evaluate, config); // flip: child mover's value + node.N[edge]++; + node.W[edge] += value; + return value; + } + + private static int SelectChild(Node node, Config config) + { + int sumN = 0; + for (int i = 0; i < node.N.Length; i++) sumN += node.N[i]; + float sqrtSum = MathF.Sqrt(sumN); + + int best = 0; + float bestScore = float.NegativeInfinity; + for (int i = 0; i < node.Moves.Length; i++) + { + float q = node.N[i] > 0 ? node.W[i] / node.N[i] : 0f; + float u = config.Cpuct * node.P[i] * sqrtSum / (1 + node.N[i]); + float score = q + u; + if (score > bestScore) { bestScore = score; best = i; } + } + return best; + } + + private static Node NewNode(IZeroSumGame game, TState state) + { + var result = game.Result(state); + if (result != GameResult.Ongoing) + return new Node { Moves = [], Terminal = true, TerminalValue = result switch { GameResult.Win => 1f, GameResult.Loss => -1f, _ => 0f } }; + return new Node { Moves = [.. game.LegalMoves(state)] }; + } + + // Evaluate the leaf, seed per-move priors (masked to legal + renormalized), and return the net's value estimate. + private static float ExpandLeaf(Node node, Evaluate evaluate, TState state) + { + var (priors, value) = evaluate(state); + int k = node.Moves.Length; + node.P = new float[k]; + node.N = new int[k]; + node.W = new float[k]; + node.Children = new Node?[k]; + + float sum = 0f; + for (int i = 0; i < k; i++) { node.P[i] = MathF.Max(priors[node.Moves[i]], 0f); sum += node.P[i]; } + if (sum > 0f) { for (int i = 0; i < k; i++) node.P[i] /= sum; } + else { for (int i = 0; i < k; i++) node.P[i] = 1f / k; } // net gave no mass to legal moves → uniform + + node.Expanded = true; + return value; + } + + private static void AddRootNoise(Node root, Config config, Xoshiro256StarStar rng) + { + if (config.RootNoiseFrac <= 0f || root.Moves.Length == 0) return; + var noise = SampleDirichlet(root.Moves.Length, config.DirichletAlpha, rng); + float frac = config.RootNoiseFrac; + for (int i = 0; i < root.P.Length; i++) root.P[i] = (1f - frac) * root.P[i] + frac * noise[i]; + } + + // ── Dirichlet sampling (Gamma via Marsaglia–Tsang; normals via Box–Muller), on the game's own RNG stream ── + private static float[] SampleDirichlet(int k, double alpha, Xoshiro256StarStar rng) + { + var g = new double[k]; + double sum = 0; + for (int i = 0; i < k; i++) { g[i] = SampleGamma(alpha, rng); sum += g[i]; } + var d = new float[k]; + if (sum <= 0) { for (int i = 0; i < k; i++) d[i] = 1f / k; return d; } + for (int i = 0; i < k; i++) d[i] = (float)(g[i] / sum); + return d; + } + + private static double SampleGamma(double alpha, Xoshiro256StarStar rng) + { + if (alpha < 1.0) + { + double u = rng.NextDouble(); + return SampleGamma(alpha + 1.0, rng) * Math.Pow(u <= 0 ? double.Epsilon : u, 1.0 / alpha); + } + double d = alpha - 1.0 / 3.0; + double c = 1.0 / Math.Sqrt(9.0 * d); + while (true) + { + double x = NextNormal(rng); + double v = 1 + c * x; + if (v <= 0) continue; + v = v * v * v; + double u2 = rng.NextDouble(); + if (u2 < 1 - 0.0331 * x * x * x * x) return d * v; + if (Math.Log(u2 <= 0 ? double.Epsilon : u2) < 0.5 * x * x + d * (1 - v + Math.Log(v))) return d * v; + } + } + + private static double NextNormal(Xoshiro256StarStar rng) + { + double u1 = rng.NextDouble(); + double u2 = rng.NextDouble(); + return Math.Sqrt(-2.0 * Math.Log(u1 <= 0 ? double.Epsilon : u1)) * Math.Cos(2.0 * Math.PI * u2); + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs new file mode 100644 index 0000000..162f5cc --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Game.cs @@ -0,0 +1,100 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +/// A Connect-4 position: 7×6 cells (0 = empty, 1 / 2 = the two players) plus the side to move. +/// Immutable from the caller's view — always returns a fresh state. +public sealed class Connect4State +{ + public const int Columns = 7; + public const int Rows = 6; + + /// Row-major, row 0 = bottom: cell (row, col) = Cells[row * Columns + col]. + public byte[] Cells { get; } + public int ToMove { get; } // 1 or 2 + + public Connect4State(byte[] cells, int toMove) { Cells = cells; ToMove = toMove; } +} + +/// +/// Connect-4 as an — the cheap first consumer of the self-play stack (tiny rules, +/// converges from random self-play in minutes on CPU, and a shallow negamax gives an exact test oracle). The action +/// index is the column (0..6); the observation is a side-to-move-relative two-plane (mine / theirs) board. +/// +public sealed class Connect4Game : IZeroSumGame +{ + public int PolicySize => Connect4State.Columns; // one action per column + public int ObservationSize => 2 * Connect4State.Columns * Connect4State.Rows; // mine + theirs planes + + public Connect4State Root(ulong? seed = null) => new(new byte[Connect4State.Columns * Connect4State.Rows], toMove: 1); + + public IReadOnlyList LegalMoves(Connect4State state) + { + var moves = new List(Connect4State.Columns); + int topRowBase = (Connect4State.Rows - 1) * Connect4State.Columns; + for (int col = 0; col < Connect4State.Columns; col++) + if (state.Cells[topRowBase + col] == 0) moves.Add(col); // top cell empty → column not full + return moves; + } + + public Connect4State Apply(Connect4State state, int move) + { + var cells = (byte[])state.Cells.Clone(); + for (int row = 0; row < Connect4State.Rows; row++) + { + int idx = row * Connect4State.Columns + move; + if (cells[idx] == 0) { cells[idx] = (byte)state.ToMove; break; } + } + return new Connect4State(cells, toMove: 3 - state.ToMove); + } + + public GameResult Result(Connect4State state) + { + int opponent = 3 - state.ToMove; + if (HasFour(state.Cells, opponent)) return GameResult.Loss; // the player who just moved completed a line + if (HasFour(state.Cells, state.ToMove)) return GameResult.Win; // defensive; unreachable in normal play + return IsFull(state.Cells) ? GameResult.Draw : GameResult.Ongoing; + } + + public void WriteObservation(Connect4State state, Span destination) + { + int n = Connect4State.Columns * Connect4State.Rows; + int mine = state.ToMove, theirs = 3 - state.ToMove; + for (int i = 0; i < n; i++) + { + destination[i] = state.Cells[i] == mine ? 1f : 0f; + destination[n + i] = state.Cells[i] == theirs ? 1f : 0f; + } + } + + private static bool IsFull(byte[] cells) + { + int topRowBase = (Connect4State.Rows - 1) * Connect4State.Columns; + for (int col = 0; col < Connect4State.Columns; col++) + if (cells[topRowBase + col] == 0) return false; + return true; + } + + // True if player p has any four-in-a-row (horizontal, vertical, or either diagonal). + private static bool HasFour(byte[] cells, int p) + { + const int C = Connect4State.Columns, R = Connect4State.Rows; + for (int row = 0; row < R; row++) + for (int col = 0; col < C; col++) + { + if (cells[row * C + col] != p) continue; + if (col + 3 < C && Line(cells, p, row, col, 0, 1)) return true; // → + if (row + 3 < R && Line(cells, p, row, col, 1, 0)) return true; // ↑ + if (row + 3 < R && col + 3 < C && Line(cells, p, row, col, 1, 1)) return true; // ↗ + if (row + 3 < R && col - 3 >= 0 && Line(cells, p, row, col, 1, -1)) return true; // ↖ + } + return false; + } + + private static bool Line(byte[] cells, int p, int row, int col, int dRow, int dCol) + { + for (int step = 1; step < 4; step++) + if (cells[(row + step * dRow) * Connect4State.Columns + (col + step * dCol)] != p) return false; + return true; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs new file mode 100644 index 0000000..e0fade2 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Connect4/Connect4Solver.cs @@ -0,0 +1,48 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +/// +/// A depth-limited negamax solver for Connect-4 — the exact oracle the self-play tests measure MCTS against (the same +/// role Kociemba plays for the cube and BFS for Rush Hour). Scores are from the side-to-move's perspective: +/// +1 = a forced win within the horizon, -1 = a forced loss, 0 = a draw or undecided within the +/// depth. Full-game solving is exponential, so callers pass a modest maxDepth and use it on tactical positions. +/// +public static class Connect4Solver +{ + private static readonly Connect4Game Game = new(); + + /// The best move for the side to move (preferring a forced win, else a draw, avoiding a loss) and its + /// negamax score in {-1, 0, +1} within plies. + public static (int Score, int BestMove) Solve(Connect4State state, int maxDepth) + { + int bestScore = int.MinValue, bestMove = -1; + foreach (int move in Game.LegalMoves(state)) + { + int score = -Negamax(Game.Apply(state, move), maxDepth - 1); + if (score > bestScore) { bestScore = score; bestMove = move; } + } + return (bestScore == int.MinValue ? 0 : bestScore, bestMove); + } + + /// The forced result of for its side to move within plies. + public static int Negamax(Connect4State state, int depth) + { + switch (Game.Result(state)) + { + case GameResult.Loss: return -1; // side to move has already lost + case GameResult.Win: return 1; + case GameResult.Draw: return 0; + } + if (depth <= 0) return 0; // undecided within the horizon → treat as neutral + + int best = -2; // below the -1..+1 range + foreach (int move in Game.LegalMoves(state)) + { + int value = -Negamax(Game.Apply(state, move), depth - 1); + if (value > best) best = value; + if (best == 1) break; // a forced win can't be improved on — prune + } + return best; + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/Connect4Tests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/Connect4Tests.cs new file mode 100644 index 0000000..3689f12 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/Connect4Tests.cs @@ -0,0 +1,93 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// +/// Connect-4 rules, the negamax oracle, and — the point of M39.1 — that is sound: with a neutral +/// (uniform-prior, zero-value) evaluator it still finds a forced win purely from tree search, agreeing with negamax. +/// This validates the novel MCTS machinery (terminal backup, zero-sum sign flip) independently of any network. +/// +public class Connect4Tests +{ + private static readonly Connect4Game Game = new(); + private const int C = Connect4State.Columns; // 7 + + // Neutral evaluator: uniform priors (all-zero → the search fills in uniform-over-legal) and value 0 — so the + // only signal MCTS can act on is the terminal outcomes it discovers by searching. + private static readonly Mcts.Evaluate Neutral = _ => (new float[C], 0f); + + private static byte[] Empty() => new byte[C * Connect4State.Rows]; + private static void Set(byte[] cells, int row, int col, int player) => cells[row * C + col] = (byte)player; + + [Fact] + public void EmptyBoard_allColumnsLegal_player1ToMove() + { + var root = Game.Root(); + Assert.Equal(1, root.ToMove); + Assert.Equal(Enumerable.Range(0, C), Game.LegalMoves(root)); + Assert.Equal(GameResult.Ongoing, Game.Result(root)); + } + + [Fact] + public void Apply_dropsToLowestEmptyRow_andFlipsSide() + { + var s1 = Game.Apply(Game.Root(), move: 3); + Assert.Equal(1, s1.Cells[0 * C + 3]); // landed on the bottom row + Assert.Equal(2, s1.ToMove); + var s2 = Game.Apply(s1, move: 3); + Assert.Equal(2, s2.Cells[1 * C + 3]); // stacked on top + Assert.Equal(1, s2.ToMove); + } + + [Fact] + public void HorizontalFour_isALossForTheSideToMove() + { + // Player 1 has a completed horizontal four on the bottom row; it is now player 2's turn to move → Loss for 2. + var cells = Empty(); + for (int col = 0; col < 4; col++) Set(cells, 0, col, 1); + var state = new Connect4State(cells, toMove: 2); + Assert.Equal(GameResult.Loss, Game.Result(state)); + } + + [Theory] + [InlineData(3)] // a mate-in-1 the winner completes by playing column 3 + public void Negamax_and_Mcts_both_find_the_mate_in_one(int winningColumn) + { + // Player 1 to move with three-in-a-row on cols 0,1,2 (bottom row); playing col 3 wins immediately. + var cells = Empty(); + Set(cells, 0, 0, 1); Set(cells, 0, 1, 1); Set(cells, 0, 2, 1); + Set(cells, 1, 0, 2); Set(cells, 1, 1, 2); // some opponent pieces, non-threatening + var state = new Connect4State(cells, toMove: 1); + + var (score, bestMove) = Connect4Solver.Solve(state, maxDepth: 1); + Assert.Equal(1, score); // a forced win for the side to move + Assert.Equal(winningColumn, bestMove); + + float[] pi = Mcts.Search(Game, state, Neutral, new Mcts.Config(Simulations: 200), new Xoshiro256StarStar(42)); + int mctsMove = Argmax(pi); + Assert.Equal(winningColumn, mctsMove); // MCTS concentrates its visits on the winning move + } + + [Fact] + public void Mcts_returns_a_distribution_over_only_legal_moves() + { + // Fill column 0 so it is illegal; MCTS must assign it zero probability. + var cells = Empty(); + for (int row = 0; row < Connect4State.Rows; row++) Set(cells, row, 0, row % 2 == 0 ? 1 : 2); + var state = new Connect4State(cells, toMove: 1); + + float[] pi = Mcts.Search(Game, state, Neutral, new Mcts.Config(Simulations: 50), new Xoshiro256StarStar(7)); + Assert.Equal(0f, pi[0]); // full column → never chosen + Assert.Equal(1f, pi.Sum(), 3); // a probability distribution + Assert.All(Game.LegalMoves(state), m => Assert.True(pi[m] >= 0f)); + } + + private static int Argmax(float[] v) + { + int best = 0; + for (int i = 1; i < v.Length; i++) if (v[i] > v[best]) best = i; + return best; + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs new file mode 100644 index 0000000..3c57662 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs @@ -0,0 +1,57 @@ +extern alias Lab; // SelfPlayCampaign is internal to the Lab exe (InternalsVisibleTo), aliased like the other campaigns + +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// +/// Contract test for the self-play campaign (PLAN M39.1): it plays self-play games, checkpoints the net + optimizer to +/// the expected store ids, and a fresh instance RESUMES the trained net rather than starting from random. Tiny sims / +/// game counts — this asserts the CONTRACT (play → checkpoint → resume), not strength. Marked Slow (it trains + touches +/// disk). Strength (win-rate climbing from random init) is the Lab `--game connect4` gate. +/// +public class SelfPlayCampaignTests +{ + private static Lab::SelfPlayCampaign Fresh() => + new(new Connect4Game(), "connect4", seed: 1, learningRate: 1e-3f, hidden: 32, + selfPlayCfg: new Mcts.Config(Simulations: 8), + gamesPerChunk: 4, tempMoves: 2, evalGames: 2, windowCapacity: 4000, maxPlies: 64); + + [Fact] + [Trait("Category", "Slow")] + public void SelfPlay_Plays_Checkpoints_AndResumesTheNet() + { + var dir = Directory.CreateTempSubdirectory("connect4-selfplay-contract"); + try + { + var store = new FileModelStore(dir.FullName); + + var c1 = Fresh(); + Assert.False(c1.Resume(store)); // empty store → fresh random net + long games1 = c1.TrainChunk(); + Assert.Equal(4, games1); // played exactly one chunk of self-play games + + var eval = c1.Evaluate(); + Assert.Contains(eval.Metrics, m => m.Name == "winRate"); + Assert.All(eval.Metrics, m => Assert.False(double.IsNaN(m.Value))); + + c1.Checkpoint(store); + c1.Dispose(); + using (var net = store.TryOpenRead("connect4", "az")) Assert.NotNull(net); // deployable net + using (var adam = store.TryOpenRead("connect4", "az-adam")) Assert.NotNull(adam); // optimizer state + + // A fresh instance must load the trained net (Resume == true) and keep training without crashing. + var c2 = Fresh(); + Assert.True(c2.Resume(store)); + long games2 = c2.TrainChunk(); + Assert.True(games2 > 0); + c2.Dispose(); + } + finally + { + dir.Delete(recursive: true); + } + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs new file mode 100644 index 0000000..4b640b4 --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs @@ -0,0 +1,31 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Environments.Connect4; + +/// +/// `--game connect4` entry point (PLAN M39.1): AlphaZero-style self-play on Connect-4 — the cheap first consumer of +/// the reusable self-play stack ( + ). Runs on the shared +/// ; the net bootstraps from random init and its win-rate vs a random-legal opponent +/// climbs. CPU-only (the net is tiny). Flags: --sims, --games, --eval-games, plus the common --hours/--data/--seed/--lr. +/// +internal static class Connect4Lab +{ + public static void Run(string[] args) + { + var a = new CliArgs(args); + double hours = a.Dbl("--hours", 1); + string dataDir = a.Str("--data", "data"); + ulong seed = a.ULong("--seed", 1); + float learningRate = a.Flt("--lr", 1e-3f); + int hidden = a.Int("--hidden", 128); // one width; the net trunk is [hidden, hidden] + int sims = a.Int("--sims", 100); // MCTS simulations per move + int gamesPerChunk = a.Int("--games", 32); + int evalGames = a.Int("--eval-games", 20); + bool evalOnly = a.Has("--eval-only"); + + var game = new Connect4Game(); + var cfg = new Mcts.Config(Simulations: sims); + LabHost.Run(args, dataDir, hours, evalOnly, useGpu: false, + _ => new SelfPlayCampaign(game, "connect4", seed, learningRate, hidden, cfg, gamesPerChunk, evalGames: evalGames), + CampaignCli.ConsoleAndCsv(Path.Combine(dataDir, "logs", "connect4-selfplay.csv"))); + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyValueTraining.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyValueTraining.cs new file mode 100644 index 0000000..b10ddeb --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyValueTraining.cs @@ -0,0 +1,34 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using Tensor = MintPlayer.AI.ReinforcementLearning.Core.Numerics.Tensor; + +/// +/// One supervised batch for a trained on soft targets — the AlphaZero loss: +/// cross-entropy of a target policy distribution π against the policy head, plus mean-squared error of a game +/// outcome z ∈ [-1,1] against the tanh of the value head. It is the same loss shape the imitation campaigns +/// use (), generalized to raw arrays + a soft π + a bounded value, so any +/// can drive it with no environment dependency. +/// +internal static class PolicyValueTraining +{ + /// Row-major observations, length ×. + /// Row-major target distributions π, length × (each row sums to 1). + /// Outcome z per row in [-1,1], length . + /// The batch's policy (CE) and value (MSE) losses. + public static (double PolicyLoss, double ValueLoss) TrainStep( + PolicyValueNet net, Adam adam, float[] obs, float[] policyTargets, float[] valueTargets, + int batch, int obsSize, int actions) + { + var (logits, value) = net.Forward(new Tensor(obs, batch, obsSize)); + var logProbs = logits.LogSoftmax(); + var ce = logProbs.Mul(new Tensor(policyTargets, batch, actions)).Sum().MulScalar(-1f / batch); + var predicted = value.Reshape(batch).Tanh(); // bound the value head to [-1,1] for a WDL outcome + var valueLoss = predicted.MseLoss(new Tensor(valueTargets, batch)); + var loss = ce.Add(valueLoss); + + adam.ZeroGrad(); + loss.Backward(); + adam.ClipGradNorm(5f); + adam.Step(); + return (ce.Data[0], valueLoss.Data[0]); + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs index ac77576..2962ad5 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs @@ -25,6 +25,7 @@ if (game.Equals("rushhour", StringComparison.OrdinalIgnoreCase)) { RushHourLab.Run(args); return; } if (game.Equals("snake", StringComparison.OrdinalIgnoreCase)) { SnakeLab.Run(args); return; } if (game.Equals("fruitcake", StringComparison.OrdinalIgnoreCase)) { FruitCakeLab.Run(args); return; } + if (game.Equals("connect4", StringComparison.OrdinalIgnoreCase)) { Connect4Lab.Run(args); return; } } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs new file mode 100644 index 0000000..7cf8039 --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs @@ -0,0 +1,277 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; +using MintPlayer.AI.ReinforcementLearning.Core.Training; + +/// +/// AlphaZero-style self-play campaign (PLAN M39) as an over any +/// . Each chunk plays a batch of games where BOTH sides are the current net guided +/// by , records (observation, MCTS visit-count π, game outcome z) into a rolling window, and +/// trains the two-headed on that window (soft-CE policy + value regression). The net +/// bootstraps from random init — no oracle, no human data. Reuses the model store + checkpoint format, the M38 +/// / plumbing, and the live-network telemetry seam. +/// +/// The game state type of . +internal sealed class SelfPlayCampaign : ITrainingCampaign, INetworkTelemetrySource +{ + private const string CheckpointKind = "selfplay-pv"; + private const string NetId = "az"; + private const string AdamId = "az-adam"; + private const int BatchSize = 128; + + private readonly IZeroSumGame _game; + private readonly string _environmentId; + private readonly float _learningRate; + private readonly int[] _hidden; + private readonly int _gamesPerChunk, _tempMoves, _evalGames, _windowCapacity, _maxPlies; + private readonly long _targetGames; + private readonly Mcts.Config _selfPlayCfg, _evalCfg; + + private readonly SeedSequence _seeds; + private readonly Xoshiro256StarStar _searchRng, _evalRng; + + private PolicyValueNet _net = null!; + private Adam _adam = null!; + private readonly List _window; + private TrainWindow _lossWindow; + private long _totalSamples, _totalGames; + private double _liveLoss = double.NaN, _lastWinRate = double.NaN; + + public SelfPlayCampaign(IZeroSumGame game, string environmentId, ulong seed, float learningRate, + int hidden, Mcts.Config selfPlayCfg, int gamesPerChunk = 32, int tempMoves = 8, int evalGames = 20, + int windowCapacity = 40_000, int maxPlies = 512, long targetGames = 0) + { + _game = game; + _environmentId = environmentId; + _learningRate = learningRate; + _hidden = [hidden, hidden]; + _selfPlayCfg = selfPlayCfg; + _evalCfg = selfPlayCfg with { RootNoiseFrac = 0f }; // eval plays deterministically (no exploration noise) + _gamesPerChunk = gamesPerChunk; + _tempMoves = tempMoves; + _evalGames = evalGames; + _windowCapacity = windowCapacity; + _maxPlies = maxPlies; + _targetGames = targetGames; + _seeds = new SeedSequence(seed); + _searchRng = _seeds.CreateRng(RngStreams.Policy); + _evalRng = _seeds.CreateRng(RngStreams.Evaluation); + _window = new List(windowCapacity); + } + + public string Environment => _environmentId; + + public bool Resume(IModelStore store) + { + bool resumed = false; + using (var s = store.TryOpenRead(_environmentId, NetId)) + { + if (s is not null) + { + _net = PolicyValueNet.Load(s, CheckpointKind, _game.ObservationSize, _game.PolicySize); + Log($"resumed {_environmentId} self-play net (trunk [{string.Join(",", _net.Trunk)}])"); + resumed = true; + } + } + if (!resumed) + { + _net = new PolicyValueNet(_game.ObservationSize, _hidden, _game.PolicySize, _seeds.CreateRng(RngStreams.Init)); + Log($"starting fresh {_environmentId} self-play (trunk [{string.Join(",", _hidden)}], {_selfPlayCfg.Simulations} sims/move)"); + } + _adam = AdamState.LoadOrInit(store, _environmentId, AdamId, _net.Parameters(), _learningRate, Log); + return resumed; + } + + public long TrainChunk() + { + for (int g = 0; g < _gamesPerChunk; g++) PlayGame(); + + // Train one shuffled pass over the current window (skip until a full batch has accumulated). + if (_window.Count >= BatchSize) + { + CubePolicyTraining.Shuffle(_window, _searchRng); + int obsSize = _game.ObservationSize, actions = _game.PolicySize; + for (int offset = 0; offset + BatchSize <= _window.Count; offset += BatchSize) + { + var obs = new float[BatchSize * obsSize]; + var pi = new float[BatchSize * actions]; + var z = new float[BatchSize]; + for (int i = 0; i < BatchSize; i++) + { + var sample = _window[offset + i]; + sample.Obs.CopyTo(obs.AsSpan(i * obsSize)); + sample.Pi.CopyTo(pi.AsSpan(i * actions)); + z[i] = sample.Z; + } + var (pl, vl) = PolicyValueTraining.TrainStep(_net, _adam, obs, pi, z, BatchSize, obsSize, actions); + _lossWindow.Add(pl, vl, 0); + _liveLoss = pl + vl; + _totalSamples += BatchSize; + } + } + return _totalGames; + } + + /// Score-maximizing: run to the runner's time budget, or an optional absolute game cap. + public bool IsComplete => _targetGames > 0 && _totalGames >= _targetGames; + + public CampaignEval Evaluate() + { + if (_net is null) return new CampaignEval([new("games", 0, "0")], "no model yet (train first)"); + + double winRate = ArenaVsRandom(); + _lastWinRate = winRate; + var (policyLoss, valueLoss, _) = _lossWindow.MeanAndReset(); + + var metrics = new List + { + new("games", _totalGames, "0"), + new("samples", _totalSamples, "0"), + new("winRate", winRate, "F3"), + new("policyLoss", policyLoss, "F4"), + new("valueLoss", valueLoss, "F4"), + }; + return new CampaignEval(metrics, + $"games {_totalGames:N0} | winRate-vs-random {winRate:P1} | policy {policyLoss:F4} | value {valueLoss:F4}"); + } + + public void Checkpoint(IModelStore store) + { + if (_net is null) return; + store.Save(_environmentId, NetId, s => _net.Save(s, CheckpointKind)); + AdamState.Save(store, _environmentId, AdamId, _adam); + } + + public void Dispose() { } + + // ── Self-play ──────────────────────────────────────────────────────────────────────────────────────────────── + private void PlayGame() + { + var state = _game.Root(); + var obsHistory = new List(64); + var piHistory = new List(64); + + int ply = 0; + while (_game.Result(state) == GameResult.Ongoing && ply < _maxPlies) + { + float[] pi = Mcts.Search(_game, state, Evaluate, _selfPlayCfg, _searchRng); + var obs = new float[_game.ObservationSize]; + _game.WriteObservation(state, obs); + obsHistory.Add(obs); + piHistory.Add(pi); + state = _game.Apply(state, SelectMove(pi, ply)); + ply++; + } + + // Assign the outcome z back through the game, alternating sign each ply (zero-sum): the terminal result is + // for the side to move in the terminal state, so the LAST position played (opposite mover) gets its negation. + float zTerminalMover = _game.Result(state) switch { GameResult.Loss => -1f, GameResult.Win => 1f, _ => 0f }; + float z = -zTerminalMover; + for (int i = obsHistory.Count - 1; i >= 0; i--) + { + AddSample(new Sample(obsHistory[i], piHistory[i], z)); + z = -z; + } + _totalGames++; + } + + private void AddSample(Sample sample) + { + if (_window.Count >= _windowCapacity) + _window.RemoveAt(0); // drop the oldest — a rolling replay window + _window.Add(sample); + } + + // MCTS leaf evaluator: masked-softmax policy priors + tanh value, read-only (no autograd). + private (float[] Priors, float Value) Evaluate(TState state) + { + var obs = new float[_game.ObservationSize]; + _game.WriteObservation(state, obs); + using (GradMode.NoGrad()) + { + var (logits, value) = _net.Forward(new Tensor(obs, 1, obs.Length)); + var priors = MaskedSoftmax(logits.Data, _game.LegalMoves(state)); + return (priors, MathF.Tanh(value.Data[0])); + } + } + + private float[] MaskedSoftmax(float[] logits, IReadOnlyList legal) + { + var priors = new float[_game.PolicySize]; + float max = float.NegativeInfinity; + foreach (int m in legal) if (logits[m] > max) max = logits[m]; + float sum = 0f; + foreach (int m in legal) { float e = MathF.Exp(logits[m] - max); priors[m] = e; sum += e; } + if (sum > 0f) foreach (int m in legal) priors[m] /= sum; + return priors; + } + + private int SelectMove(float[] pi, int ply) + { + if (ply >= _tempMoves) // late game: play the most-visited move + { + int best = 0; + for (int a = 1; a < pi.Length; a++) if (pi[a] > pi[best]) best = a; + return best; + } + // early game: sample proportional to visit counts, for opening variety + double r = _searchRng.NextDouble(), acc = 0; + for (int a = 0; a < pi.Length; a++) { acc += pi[a]; if (r <= acc && pi[a] > 0) return a; } + // numerical fallback: the last legal (highest-visited) move + int fallback = 0; + for (int a = 1; a < pi.Length; a++) if (pi[a] > pi[fallback]) fallback = a; + return fallback; + } + + // ── Eval: the model (net + MCTS, no root noise, argmax) vs a random-legal opponent, colors alternated ── + private double ArenaVsRandom() + { + double score = 0; // win = 1, draw = 0.5 + for (int game = 0; game < _evalGames; game++) + { + int modelSide = game % 2 == 0 ? 1 : 2; // model is player 1 on even games, player 2 on odd + var state = _game.Root(); + int mover = 1, ply = 0; + GameResult result; + while ((result = _game.Result(state)) == GameResult.Ongoing && ply < _maxPlies) + { + int move = mover == modelSide ? ModelMove(state) : RandomLegalMove(state); + state = _game.Apply(state, move); + mover = 3 - mover; + ply++; + } + // result is for the side to move in the terminal state (who did NOT just move). Loss ⇒ the last mover won. + if (result == GameResult.Loss) { int lastMover = 3 - mover; if (lastMover == modelSide) score += 1; } + else if (result != GameResult.Win) score += 0.5; // draw (ply cap or full board) + } + return score / _evalGames; + } + + private int ModelMove(TState state) + { + float[] pi = Mcts.Search(_game, state, Evaluate, _evalCfg, _evalRng); + int best = 0; + for (int a = 1; a < pi.Length; a++) if (pi[a] > pi[best]) best = a; + return best; + } + + private int RandomLegalMove(TState state) + { + var legal = _game.LegalMoves(state); + return legal[_evalRng.NextInt(legal.Count)]; + } + + // ── Live telemetry (INetworkTelemetrySource): read-only snapshot of the current net ── + string INetworkTelemetrySource.NetKind => "policy-value"; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => _net is null ? null : [.. _net.Parameters()]; + NetworkMetrics INetworkTelemetrySource.Sample() + => new(_totalGames, _targetGames, _liveLoss, _lastWinRate, double.NaN); + + private static void Log(string message) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} {message}"); + + private sealed record Sample(float[] Obs, float[] Pi, float Z); +} From bd401cdea6ed1eff3f3fb68662c8439971d88ac5 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 21:58:14 +0200 Subject: [PATCH 03/33] M39.2: chess as the self-play stack's second consumer (perft-verified) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Chess plugs into the M39.1 rails — SelfPlayCampaign, MCTS, PolicyValueNet, and the training step are REUSED UNCHANGED. Only chess-specific code is new: - Environments/Chess/ChessBoard.cs (ChessRules + ChessState) — full legal move generation (castling through/into check, en passant, promotion, pins/checks) + make-move + attack detection + terminal detection (checkmate, stalemate, 50-move, insufficient material). Correctness-first mailbox engine. - ChessFen.cs — FEN parse/render for test positions. - ChessMoveEncoding.cs — the AlphaZero 64×73 = 4672 action space (queen/knight/ underpromotion planes); encode + decode (queen-promo inferred on apply). - ChessGame : IZeroSumGame — 18-plane observation, legal-moves→ indices, apply-by-index. - Lab/ChessLab.cs + --game chess dispatch. Gates met: - PERFT (the hard movegen gate): 25/25 published node counts match, incl. startpos depth 5 = 4,865,609 and Kiwipete depth 4 = 4,085,603 (3s Release). - Move encoding: encode→decode→Apply round-trips every legal move on positions rich in castling/en-passant/all promotion flavours, no index collisions; Result detects fool's-mate and stalemate. - End-to-end: a 2-minute `--game chess` run from random init plays legal chess and reaches 62.5% vs a random-legal opponent (16 sims/move), reusing the rails. - 355 fast tests green (+24 chess). Threefold-repetition, perspective canonicalization, batched-leaf MCTS, and the anti-exploitation league/opening-diversity levers (see the PRD robustness note added here) are M39.3. Honest scope: MLP-over-flattened-planes → legal, improving play, not engine strength. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/CHESS_SELFPLAY_PRD.md | 1 + docs/prd/PLAN.md | 9 +- .../Chess/ChessBoard.cs | 350 ++++++++++++++++++ .../Chess/ChessFen.cs | 101 +++++ .../Chess/ChessGame.cs | 71 ++++ .../Chess/ChessMoveEncoding.cs | 96 +++++ .../ChessEncodingTests.cs | 53 +++ .../ChessPerftTests.cs | 59 +++ .../ChessLab.cs | 33 ++ .../Program.cs | 1 + 10 files changed, 772 insertions(+), 2 deletions(-) create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs create mode 100644 tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessEncodingTests.cs create mode 100644 tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessPerftTests.cs create mode 100644 tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs diff --git a/docs/prd/CHESS_SELFPLAY_PRD.md b/docs/prd/CHESS_SELFPLAY_PRD.md index 9df073b..00bdc1e 100644 --- a/docs/prd/CHESS_SELFPLAY_PRD.md +++ b/docs/prd/CHESS_SELFPLAY_PRD.md @@ -103,6 +103,7 @@ public static class Mcts 2. **MCTS soundness / self-play collapse.** Wrong value-sign backup, missing Dirichlet root noise, or no temperature schedule → collapsing targets. Mitigation: unit-test MCTS on forced-win/forced-draw Connect-4 positions against negamax; gate training on rising win-rate vs a *frozen* baseline. 3. **Compute.** Self-play games are long; MCTS is hundreds of sims/move. CPU-bound (the small MLP won't clear the GPU threshold). Mitigation: Connect-4 for the convergence proof; keep chess sims modest; batched-leaf MCTS is a phase-3 lever. Set expectations: legal, improving play — not engine strength. 4. **Representation ceiling.** MLP over flattened planes has no spatial prior. Accepted for v1; conv backend is a separate workstream if positional strength becomes a goal. +5. **Robustness / exploitability — the "a novel move disorients the AI" failure (M39.3 lever).** Two nets trained only against each other co-adapt into a narrow shared distribution; off-distribution positions (a human's weird/suboptimal move) are undertrained, so the value/policy net misjudges them and — with too few sims — the agent can lose to a much weaker player. This is real even at superhuman level (the KataGo cyclic-group exploit let amateurs beat a superhuman Go bot). **Primary defence is already in the design:** at inference the agent runs **MCTS from the actual position** (it computes, it doesn't recall), so it generalizes to novel positions far better than a reactive net — this is a core reason we chose AlphaZero over reactive self-play. **Coverage levers (M39.3):** (a) **Dirichlet root noise + temperature sampling** — *already in the M39.1 MCTS + campaign* — make self-play explore off-policy lines so the training distribution doesn't collapse; (b) **opponent-pool / league play** — train against a pool of past checkpoints and occasionally a **random/weak mover**, the direct fix for "punish an unexpected blunder" and for co-adaptation; (c) **diverse/randomized opening positions**; (d) enough inference sims + **adversarial fine-tune** on any discovered exploit. Note: **NoisyNets is *not* the right tool here** — it's a DQN weight-noise exploration trick that perturbs the policy globally but does not ensure the agent *visits* diverse positions; AlphaZero exploration is Dirichlet-noise + temperature + opponent diversity, which target coverage directly. ## 7. Verification diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 9bd36de..6813f8c 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1390,8 +1390,13 @@ self-play loop, and each game's rules. + `--game chess` Lab dispatch; an Elo eval (`ChessAb`, mirroring `FruitCakeAb`). **Gate (the hard one): perft** node-counts matched to published values (startpos, Kiwipete, …) to depth 5–6 *before any training*; move encode/decode round-trips; terminated-vs-truncated split correct; win-rate vs random-legal climbs; contract-test resume; ship `models/chess.az.ckpt` (LFS). -- **M39.3 — scale (optional).** Batched-leaf MCTS; GPU-resident forward *if* the net grows enough to clear the routing threshold - (honest: a small chess MLP won't); conv-backend support for positional strength (a separate backend workstream); a web showcase page. +- **M39.3 — scale + robustness (optional).** Batched-leaf MCTS; GPU-resident forward *if* the net grows enough to clear the + routing threshold (honest: a small chess MLP won't); conv-backend support for positional strength (a separate backend workstream); + a web showcase page. **Anti-exploitation levers** (so a novel/weak human move can't disorient the net into losing — real even at + superhuman level, cf. the KataGo cyclic-group exploit): opponent-**pool/league** play + an occasional random/weak mover, diverse + randomized opening positions, and adversarial fine-tune on any discovered exploit. (Dirichlet root noise + temperature already ship + in the M39.1 MCTS/campaign; search-from-the-actual-position is the primary defence. NoisyNets is *not* the right tool — it perturbs + the policy globally rather than broadening position coverage.) **Honest scope.** MLP-only net (no conv) over a flattened board → *legal, steadily-improving* play, not engine strength; self-play is CPU-bound (the small MLP won't hit the GPU lever that helps the cube). Connect-4 is where the self-improvement curve is unmistakable; diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs new file mode 100644 index 0000000..ea1e37b --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs @@ -0,0 +1,350 @@ +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// Piece type (colour-agnostic). = empty square. +public enum PieceType : byte { None = 0, Pawn = 1, Knight = 2, Bishop = 3, Rook = 4, Queen = 5, King = 6 } + +/// A move: from/to squares (0..63, = rank*8 + file, a1 = 0), plus the promotion piece for a pawn reaching +/// the last rank ( otherwise). Castling, en passant, and double-push are inferred from +/// the board when the move is made, so they need no extra flag here. +public readonly record struct ChessMove(byte From, byte To, PieceType Promotion = PieceType.None); + +/// +/// A chess position (mailbox board + side to move + castling rights + en-passant target + halfmove clock) and the +/// full rules: legal move generation (castling, en passant, promotion, pins/checks), make-move, attack detection, +/// and terminal detection (checkmate, stalemate, 50-move, insufficient material). Correctness-first (clone-per-move, +/// no bitboards); verified by perft. Squares are 0..63 with file = sq & 7, rank = sq >> 3; White moves +/// toward higher ranks. Board cells: 0 = empty, +1..+6 = White P,N,B,R,Q,K, −1..−6 = Black (sign = colour). +/// Threefold-repetition is intentionally not modelled in v1 (see PLAN M39); the 50-move rule and the self-play +/// ply cap bound looping games. +/// +public sealed class ChessState +{ + public sbyte[] Squares { get; } // length 64 + public bool WhiteToMove { get; } + public byte Castling { get; } // bit 0 = White O-O, 1 = White O-O-O, 2 = Black O-O, 3 = Black O-O-O + public sbyte EnPassant { get; } // target square a pawn could capture onto, or -1 + public byte HalfmoveClock { get; } + + public const byte CastleWK = 1, CastleWQ = 2, CastleBK = 4, CastleBQ = 8; + + public ChessState(sbyte[] squares, bool whiteToMove, byte castling, sbyte enPassant, byte halfmoveClock) + { + Squares = squares; + WhiteToMove = whiteToMove; + Castling = castling; + EnPassant = enPassant; + HalfmoveClock = halfmoveClock; + } + + public static ChessState StartPosition() => ChessFen.Parse(ChessFen.StartFen); +} + +/// Static chess rules over . +public static class ChessRules +{ + private static int File(int sq) => sq & 7; + private static int Rank(int sq) => sq >> 3; + private static PieceType Type(sbyte piece) => (PieceType)Math.Abs(piece); + private static bool IsWhite(sbyte piece) => piece > 0; + + private static readonly (int df, int dr)[] KnightDeltas = + [(1, 2), (2, 1), (2, -1), (1, -2), (-1, -2), (-2, -1), (-2, 1), (-1, 2)]; + private static readonly (int df, int dr)[] KingDeltas = + [(1, 0), (1, 1), (0, 1), (-1, 1), (-1, 0), (-1, -1), (0, -1), (1, -1)]; + private static readonly (int df, int dr)[] BishopDirs = [(1, 1), (-1, 1), (1, -1), (-1, -1)]; + private static readonly (int df, int dr)[] RookDirs = [(1, 0), (-1, 0), (0, 1), (0, -1)]; + + // ── Attack detection ──────────────────────────────────────────────────────────────────────────────────────── + + /// Whether is attacked by the given colour's pieces. + public static bool IsSquareAttacked(ChessState s, int square, bool byWhite) + { + int tf = File(square), tr = Rank(square); + var b = s.Squares; + + // Pawns: a pawn on the attacking side attacks diagonally toward its forward direction. + int pawnDir = byWhite ? 1 : -1; + sbyte pawn = (sbyte)(byWhite ? 1 : -1); + foreach (int df in stackalloc[] { -1, 1 }) + { + int f = tf + df, r = tr - pawnDir; // a pawn on (r) attacking (tr) sits one rank behind in its dir + if ((uint)f < 8 && (uint)r < 8 && b[r * 8 + f] == pawn) return true; + } + + foreach (var (df, dr) in KnightDeltas) + { + int f = tf + df, r = tr + dr; + if ((uint)f < 8 && (uint)r < 8 && b[r * 8 + f] == (sbyte)(byWhite ? 2 : -2)) return true; + } + + foreach (var (df, dr) in KingDeltas) + { + int f = tf + df, r = tr + dr; + if ((uint)f < 8 && (uint)r < 8 && b[r * 8 + f] == (sbyte)(byWhite ? 6 : -6)) return true; + } + + // Sliders: bishops/queens on diagonals, rooks/queens on ranks/files. + if (RayHits(b, tf, tr, BishopDirs, byWhite, PieceType.Bishop)) return true; + if (RayHits(b, tf, tr, RookDirs, byWhite, PieceType.Rook)) return true; + return false; + } + + private static bool RayHits(sbyte[] b, int tf, int tr, (int df, int dr)[] dirs, bool byWhite, PieceType slider) + { + foreach (var (df, dr) in dirs) + { + int f = tf + df, r = tr + dr; + while ((uint)f < 8 && (uint)r < 8) + { + sbyte piece = b[r * 8 + f]; + if (piece != 0) + { + if (IsWhite(piece) == byWhite) + { + var t = Type(piece); + if (t == slider || t == PieceType.Queen) return true; + } + break; // blocked + } + f += df; r += dr; + } + } + return false; + } + + public static int KingSquare(ChessState s, bool white) + { + sbyte king = (sbyte)(white ? 6 : -6); + for (int sq = 0; sq < 64; sq++) if (s.Squares[sq] == king) return sq; + return -1; + } + + public static bool InCheck(ChessState s, bool white) + => IsSquareAttacked(s, KingSquare(s, white), byWhite: !white); + + // ── Move generation ───────────────────────────────────────────────────────────────────────────────────────── + + /// All fully-legal moves for the side to move (own king not left in check). + public static List LegalMoves(ChessState s) + { + var pseudo = PseudoLegal(s); + var legal = new List(pseudo.Count); + foreach (var move in pseudo) + { + var next = MakeMove(s, move); + if (!InCheck(next, white: s.WhiteToMove)) legal.Add(move); // the side that just moved must be safe + } + return legal; + } + + private static List PseudoLegal(ChessState s) + { + var moves = new List(48); + bool white = s.WhiteToMove; + var b = s.Squares; + + for (int sq = 0; sq < 64; sq++) + { + sbyte piece = b[sq]; + if (piece == 0 || IsWhite(piece) != white) continue; + int f = File(sq), r = Rank(sq); + switch (Type(piece)) + { + case PieceType.Pawn: PawnMoves(s, sq, f, r, white, moves); break; + case PieceType.Knight: StepMoves(b, sq, f, r, white, KnightDeltas, moves); break; + case PieceType.King: StepMoves(b, sq, f, r, white, KingDeltas, moves); CastleMoves(s, white, moves); break; + case PieceType.Bishop: SlideMoves(b, sq, f, r, white, BishopDirs, moves); break; + case PieceType.Rook: SlideMoves(b, sq, f, r, white, RookDirs, moves); break; + case PieceType.Queen: + SlideMoves(b, sq, f, r, white, BishopDirs, moves); + SlideMoves(b, sq, f, r, white, RookDirs, moves); + break; + } + } + return moves; + } + + private static void StepMoves(sbyte[] b, int sq, int f, int r, bool white, (int df, int dr)[] deltas, List moves) + { + foreach (var (df, dr) in deltas) + { + int nf = f + df, nr = r + dr; + if ((uint)nf >= 8 || (uint)nr >= 8) continue; + int to = nr * 8 + nf; + sbyte target = b[to]; + if (target == 0 || IsWhite(target) != white) moves.Add(new ChessMove((byte)sq, (byte)to)); + } + } + + private static void SlideMoves(sbyte[] b, int sq, int f, int r, bool white, (int df, int dr)[] dirs, List moves) + { + foreach (var (df, dr) in dirs) + { + int nf = f + df, nr = r + dr; + while ((uint)nf < 8 && (uint)nr < 8) + { + int to = nr * 8 + nf; + sbyte target = b[to]; + if (target == 0) moves.Add(new ChessMove((byte)sq, (byte)to)); + else { if (IsWhite(target) != white) moves.Add(new ChessMove((byte)sq, (byte)to)); break; } + nf += df; nr += dr; + } + } + } + + private static void PawnMoves(ChessState s, int sq, int f, int r, bool white, List moves) + { + var b = s.Squares; + int dir = white ? 1 : -1; + int startRank = white ? 1 : 6; + int lastRank = white ? 7 : 0; + + // Single / double push. + int one = (r + dir) * 8 + f; + if (b[one] == 0) + { + AddPawn(moves, sq, one, r + dir == lastRank); + if (r == startRank) + { + int two = (r + 2 * dir) * 8 + f; + if (b[two] == 0) moves.Add(new ChessMove((byte)sq, (byte)two)); + } + } + + // Captures (incl. en passant). + foreach (int df in stackalloc[] { -1, 1 }) + { + int nf = f + df, nr = r + dir; + if ((uint)nf >= 8 || (uint)nr >= 8) continue; + int to = nr * 8 + nf; + sbyte target = b[to]; + if (target != 0 && IsWhite(target) != white) AddPawn(moves, sq, to, nr == lastRank); + else if (to == s.EnPassant) moves.Add(new ChessMove((byte)sq, (byte)to)); // en passant onto the empty ep square + } + } + + private static void AddPawn(List moves, int from, int to, bool promotion) + { + if (promotion) + { + moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Queen)); + moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Rook)); + moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Bishop)); + moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Knight)); + } + else moves.Add(new ChessMove((byte)from, (byte)to)); + } + + private static void CastleMoves(ChessState s, bool white, List moves) + { + var b = s.Squares; + int rank = white ? 0 : 7; + int kingSq = rank * 8 + 4; + if (b[kingSq] != (sbyte)(white ? 6 : -6)) return; + if (IsSquareAttacked(s, kingSq, byWhite: !white)) return; // can't castle out of check + + byte kSide = white ? ChessState.CastleWK : ChessState.CastleBK; + byte qSide = white ? ChessState.CastleWQ : ChessState.CastleBQ; + + // King-side: squares f,g empty; king doesn't pass through/into attack. + if ((s.Castling & kSide) != 0 && b[rank * 8 + 5] == 0 && b[rank * 8 + 6] == 0 + && !IsSquareAttacked(s, rank * 8 + 5, !white) && !IsSquareAttacked(s, rank * 8 + 6, !white)) + moves.Add(new ChessMove((byte)kingSq, (byte)(rank * 8 + 6))); + + // Queen-side: squares b,c,d empty; king passes over d,c (b need not be safe, only empty). + if ((s.Castling & qSide) != 0 && b[rank * 8 + 1] == 0 && b[rank * 8 + 2] == 0 && b[rank * 8 + 3] == 0 + && !IsSquareAttacked(s, rank * 8 + 3, !white) && !IsSquareAttacked(s, rank * 8 + 2, !white)) + moves.Add(new ChessMove((byte)kingSq, (byte)(rank * 8 + 2))); + } + + // ── Make move ─────────────────────────────────────────────────────────────────────────────────────────────── + + /// The position after (a fresh state; is unchanged). + /// Castling / en passant / promotion / double-push are inferred from the board. + public static ChessState MakeMove(ChessState s, ChessMove move) + { + var b = (sbyte[])s.Squares.Clone(); + bool white = s.WhiteToMove; + sbyte piece = b[move.From]; + var type = Type(piece); + bool capture = b[move.To] != 0; + sbyte newEp = -1; + byte castling = s.Castling; + + b[move.From] = 0; + + // En passant capture: the taken pawn sits beside the destination, not on it. + if (type == PieceType.Pawn && move.To == s.EnPassant && s.EnPassant >= 0) + { + int capturedSq = move.To + (white ? -8 : 8); + b[capturedSq] = 0; + capture = true; + } + + // Place the piece (promotion swaps in the chosen piece). + b[move.To] = move.Promotion != PieceType.None + ? (sbyte)((white ? 1 : -1) * (int)move.Promotion) + : piece; + + // Double push sets the en-passant target square. + if (type == PieceType.Pawn && Math.Abs(Rank(move.To) - Rank(move.From)) == 2) + newEp = (sbyte)((move.From + move.To) / 2); + + // Castling: move the rook too. + if (type == PieceType.King && Math.Abs(File(move.To) - File(move.From)) == 2) + { + int rank = Rank(move.From); + if (File(move.To) == 6) { b[rank * 8 + 5] = b[rank * 8 + 7]; b[rank * 8 + 7] = 0; } // king-side + else { b[rank * 8 + 3] = b[rank * 8 + 0]; b[rank * 8 + 0] = 0; } // queen-side + } + + // Update castling rights: king move clears both; rook move/capture clears that corner. + if (type == PieceType.King) castling &= (byte)(white ? ~(ChessState.CastleWK | ChessState.CastleWQ) : ~(ChessState.CastleBK | ChessState.CastleBQ)); + castling &= CastlingMask(move.From); + castling &= CastlingMask(move.To); // a rook captured on its home square loses that right too + + byte halfmove = (byte)(type == PieceType.Pawn || capture ? 0 : s.HalfmoveClock + 1); + return new ChessState(b, !white, castling, newEp, halfmove); + } + + // Clears the castling right whose rook/king home square is 'sq' (identity mask otherwise). + private static byte CastlingMask(int sq) => sq switch + { + 0 => unchecked((byte)~ChessState.CastleWQ), // a1 rook + 7 => unchecked((byte)~ChessState.CastleWK), // h1 rook + 56 => unchecked((byte)~ChessState.CastleBQ), // a8 rook + 63 => unchecked((byte)~ChessState.CastleBK), // h8 rook + _ => 0xFF, + }; + + // ── Terminal detection ────────────────────────────────────────────────────────────────────────────────────── + + public static bool IsFiftyMove(ChessState s) => s.HalfmoveClock >= 100; + + public static bool IsInsufficientMaterial(ChessState s) + { + int knights = 0, bishops = 0; + foreach (sbyte p in s.Squares) + { + switch (Type(p)) + { + case PieceType.None or PieceType.King: break; + case PieceType.Knight: knights++; break; + case PieceType.Bishop: bishops++; break; + default: return false; // a pawn/rook/queen = enough material + } + } + return knights + bishops <= 1; // K vs K, K+N vs K, K+B vs K (not exhaustive, but the common draws) + } + + /// Perft: the number of leaf nodes at — the movegen correctness oracle. + public static long Perft(ChessState s, int depth) + { + if (depth == 0) return 1; + var moves = LegalMoves(s); + if (depth == 1) return moves.Count; + long nodes = 0; + foreach (var move in moves) nodes += Perft(MakeMove(s, move), depth - 1); + return nodes; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs new file mode 100644 index 0000000..23599ba --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessFen.cs @@ -0,0 +1,101 @@ +using System.Text; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// Forsyth–Edwards Notation (FEN) ↔ . Used to set up standard test positions (perft) and to +/// render a position for debugging. Only the fields the engine tracks are parsed (board, side, castling, en passant, +/// halfmove clock); the fullmove number is ignored on read and written as 1. +/// +public static class ChessFen +{ + public const string StartFen = "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"; + + public static ChessState Parse(string fen) + { + string[] parts = fen.Split(' '); + var squares = new sbyte[64]; + + string[] ranks = parts[0].Split('/'); // rank 8 first + for (int i = 0; i < 8; i++) + { + int rank = 7 - i; // ranks[0] is the 8th rank (rank index 7) + int file = 0; + foreach (char c in ranks[i]) + { + if (char.IsDigit(c)) { file += c - '0'; continue; } + squares[rank * 8 + file] = PieceOf(c); + file++; + } + } + + bool whiteToMove = parts[1] == "w"; + byte castling = 0; + if (parts[2] != "-") + foreach (char c in parts[2]) + castling |= c switch + { + 'K' => ChessState.CastleWK, + 'Q' => ChessState.CastleWQ, + 'k' => ChessState.CastleBK, + 'q' => ChessState.CastleBQ, + _ => (byte)0, + }; + + sbyte enPassant = parts[3] == "-" ? (sbyte)-1 : (sbyte)SquareOf(parts[3]); + byte halfmove = parts.Length > 4 && byte.TryParse(parts[4], out byte hm) ? hm : (byte)0; + return new ChessState(squares, whiteToMove, castling, enPassant, halfmove); + } + + public static string ToFen(ChessState s) + { + var sb = new StringBuilder(); + for (int rank = 7; rank >= 0; rank--) + { + int empty = 0; + for (int file = 0; file < 8; file++) + { + sbyte piece = s.Squares[rank * 8 + file]; + if (piece == 0) { empty++; continue; } + if (empty > 0) { sb.Append(empty); empty = 0; } + sb.Append(CharOf(piece)); + } + if (empty > 0) sb.Append(empty); + if (rank > 0) sb.Append('/'); + } + sb.Append(s.WhiteToMove ? " w " : " b "); + string rights = string.Concat( + (s.Castling & ChessState.CastleWK) != 0 ? "K" : "", + (s.Castling & ChessState.CastleWQ) != 0 ? "Q" : "", + (s.Castling & ChessState.CastleBK) != 0 ? "k" : "", + (s.Castling & ChessState.CastleBQ) != 0 ? "q" : ""); + sb.Append(rights.Length == 0 ? "-" : rights); + sb.Append(' ').Append(s.EnPassant < 0 ? "-" : SquareName(s.EnPassant)); + sb.Append(' ').Append(s.HalfmoveClock).Append(" 1"); + return sb.ToString(); + } + + private static sbyte PieceOf(char c) + { + sbyte type = char.ToLowerInvariant(c) switch + { + 'p' => 1, 'n' => 2, 'b' => 3, 'r' => 4, 'q' => 5, 'k' => 6, + _ => throw new FormatException($"Bad FEN piece '{c}'."), + }; + return char.IsUpper(c) ? type : (sbyte)-type; + } + + private static char CharOf(sbyte piece) + { + char c = (PieceType)Math.Abs(piece) switch + { + PieceType.Pawn => 'p', PieceType.Knight => 'n', PieceType.Bishop => 'b', + PieceType.Rook => 'r', PieceType.Queen => 'q', PieceType.King => 'k', + _ => '?', + }; + return piece > 0 ? char.ToUpperInvariant(c) : c; + } + + private static int SquareOf(string s) => (s[1] - '1') * 8 + (s[0] - 'a'); + private static string SquareName(int sq) => $"{(char)('a' + (sq & 7))}{(char)('1' + (sq >> 3))}"; +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs new file mode 100644 index 0000000..58ef92c --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs @@ -0,0 +1,71 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; + +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// Chess as an — the headline consumer of the self-play stack (PLAN M39.2). It +/// adapts the perft-verified engine to the seam: the action space is the 4672-index +/// , and the observation is an 18-plane board stack (12 piece planes + side-to-move + +/// 4 castling-rights + en passant), flattened. MCTS, the two-headed net, and the self-play campaign are all reused +/// unchanged from M39.1. (Board geometry is absolute — the net learns both colours via the side-to-move plane; +/// perspective canonicalization is a later efficiency lever, PLAN M39.3.) +/// +public sealed class ChessGame : IZeroSumGame +{ + private const int Planes = 18; + + public int PolicySize => ChessMoveEncoding.Size; // 4672 + public int ObservationSize => Planes * 64; // 1152 + + public ChessState Root(ulong? seed = null) => ChessState.StartPosition(); + + public IReadOnlyList LegalMoves(ChessState state) + { + var moves = ChessRules.LegalMoves(state); + var indices = new int[moves.Count]; + for (int i = 0; i < moves.Count; i++) indices[i] = ChessMoveEncoding.Encode(moves[i]); + return indices; + } + + public ChessState Apply(ChessState state, int move) + { + var decoded = ChessMoveEncoding.Decode(move); + // A queen-promotion rides the queen planes and decodes as None → promote the pawn to a Queen by default. + if (decoded.Promotion == PieceType.None + && (PieceType)Math.Abs(state.Squares[decoded.From]) == PieceType.Pawn + && (decoded.To >> 3) is 0 or 7) + decoded = decoded with { Promotion = PieceType.Queen }; + return ChessRules.MakeMove(state, decoded); + } + + public GameResult Result(ChessState state) + { + if (ChessRules.LegalMoves(state).Count == 0) + return ChessRules.InCheck(state, state.WhiteToMove) ? GameResult.Loss : GameResult.Draw; // mate vs stalemate + if (ChessRules.IsFiftyMove(state) || ChessRules.IsInsufficientMaterial(state)) + return GameResult.Draw; + return GameResult.Ongoing; + } + + public void WriteObservation(ChessState state, Span destination) + { + destination.Clear(); + // Piece planes 0..5 = White P,N,B,R,Q,K ; 6..11 = Black. Plane p, square sq → index p*64 + sq. + for (int sq = 0; sq < 64; sq++) + { + sbyte piece = state.Squares[sq]; + if (piece == 0) continue; + int type = Math.Abs(piece) - 1; // 0..5 + int plane = piece > 0 ? type : 6 + type; // white vs black block + destination[plane * 64 + sq] = 1f; + } + if (state.WhiteToMove) Fill(destination, 12); + if ((state.Castling & ChessState.CastleWK) != 0) Fill(destination, 13); + if ((state.Castling & ChessState.CastleWQ) != 0) Fill(destination, 14); + if ((state.Castling & ChessState.CastleBK) != 0) Fill(destination, 15); + if ((state.Castling & ChessState.CastleBQ) != 0) Fill(destination, 16); + if (state.EnPassant >= 0) destination[17 * 64 + state.EnPassant] = 1f; + } + + private static void Fill(Span dest, int plane) => dest.Slice(plane * 64, 64).Fill(1f); +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs new file mode 100644 index 0000000..f7ddea4 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs @@ -0,0 +1,96 @@ +namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// The AlphaZero move encoding: a fixed 64 × 73 = 4672 action space (from-square × move-type plane), which is +/// the policy-head width. The 73 planes are 56 "queen" moves (8 directions × 7 distances — this also carries +/// queen-promotions and every sliding/king/pawn move), 8 knight moves, and 9 underpromotions (Knight/Bishop/Rook × +/// {capture-left, straight, capture-right}). Distinct legal moves always map to distinct indices, so the search can +/// tell them apart; queen-promotions ride the queen planes (decoded promotion is inferred as Queen when a pawn +/// reaches the last rank — see ). +/// +public static class ChessMoveEncoding +{ + public const int PlanesPerSquare = 73; + public const int Size = 64 * PlanesPerSquare; // 4672 + + // 8 sliding directions (df, dr), indices 0..7 — shared by encode and decode. + private static readonly (int df, int dr)[] Dirs = + [(0, 1), (1, 1), (1, 0), (1, -1), (0, -1), (-1, -1), (-1, 0), (-1, 1)]; + // 8 knight deltas (df, dr), indices 0..7. + private static readonly (int df, int dr)[] Knights = + [(1, 2), (2, 1), (2, -1), (1, -2), (-1, -2), (-2, -1), (-2, 1), (-1, 2)]; + + private static int File(int sq) => sq & 7; + private static int Rank(int sq) => sq >> 3; + + /// The action index (0..4671) for a legal move. Queen-promotions use the queen planes; N/B/R + /// promotions use the underpromotion planes. + public static int Encode(ChessMove move) + { + int df = File(move.To) - File(move.From); + int dr = Rank(move.To) - Rank(move.From); + int plane; + + if (move.Promotion is PieceType.Knight or PieceType.Bishop or PieceType.Rook) + { + int pieceIdx = move.Promotion switch { PieceType.Knight => 0, PieceType.Bishop => 1, _ => 2 }; + plane = 64 + pieceIdx * 3 + (df + 1); // df ∈ {-1,0,1} → {capture-left, straight, capture-right} + } + else if (IsKnight(df, dr)) + { + plane = 56 + KnightIndex(df, dr); + } + else + { + int dir = DirIndex(Math.Sign(df), Math.Sign(dr)); + int dist = Math.Max(Math.Abs(df), Math.Abs(dr)); + plane = dir * 7 + (dist - 1); + } + return move.From * PlanesPerSquare + plane; + } + + /// Decodes an action index to a move. A queen-promotion decodes with — the + /// caller promotes a pawn reaching the last rank to a Queen by default. + public static ChessMove Decode(int index) + { + int from = index / PlanesPerSquare, plane = index % PlanesPerSquare; + int f = File(from), r = Rank(from); + + if (plane >= 64) + { + int p = plane - 64; + var promo = (p / 3) switch { 0 => PieceType.Knight, 1 => PieceType.Bishop, _ => PieceType.Rook }; + int df = (p % 3) - 1; + int dr = r == 6 ? 1 : -1; // a pawn on the 7th rank promotes upward, on the 2nd rank downward + return new ChessMove((byte)from, (byte)((r + dr) * 8 + (f + df)), promo); + } + if (plane >= 56) + { + var (df, dr) = Knights[plane - 56]; + return new ChessMove((byte)from, (byte)((r + dr) * 8 + (f + df))); + } + { + var (df, dr) = Dirs[plane / 7]; + int dist = plane % 7 + 1; + return new ChessMove((byte)from, (byte)((r + dr * dist) * 8 + (f + df * dist))); + } + } + + private static bool IsKnight(int df, int dr) + { + int a = Math.Abs(df), b = Math.Abs(dr); + return (a == 1 && b == 2) || (a == 2 && b == 1); + } + + private static int KnightIndex(int df, int dr) + { + for (int i = 0; i < Knights.Length; i++) if (Knights[i] == (df, dr)) return i; + throw new ArgumentException($"Not a knight move: ({df},{dr})."); + } + + private static int DirIndex(int sdf, int sdr) + { + for (int i = 0; i < Dirs.Length; i++) if (Dirs[i] == (sdf, sdr)) return i; + throw new ArgumentException($"Not a straight/diagonal direction: ({sdf},{sdr})."); + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessEncodingTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessEncodingTests.cs new file mode 100644 index 0000000..fedd839 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessEncodingTests.cs @@ -0,0 +1,53 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// +/// The 4672-index move encoding (PLAN M39.2): distinct legal moves must map to distinct indices (so search can tell +/// them apart), and encode → decode → must reproduce exactly the position that making +/// the move directly produces — the end-to-end check that the policy indices, the decoder, and promotion-inference +/// all agree. Run over positions rich in castling, en passant, and every promotion flavour. +/// +public class ChessEncodingTests +{ + [Theory] + [InlineData(ChessFen.StartFen)] + [InlineData("r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1")] // Kiwipete + [InlineData("4k3/P7/8/8/8/8/8/4K3 w - - 0 1")] // a7-a8 promotion push (Q/R/B/N) + [InlineData("1n2k3/P7/8/8/8/8/8/4K3 w - - 0 1")] // a7xb8 promotion by capture (Q/R/B/N) and the push + [InlineData("4k3/8/8/8/8/8/p7/4K3 b - - 0 1")] // a black promotion (downward direction) + public void Encode_decode_apply_roundtrips_every_legal_move(string fen) + { + var game = new ChessGame(); + var state = ChessFen.Parse(fen); + var legal = ChessRules.LegalMoves(state); + var seen = new HashSet(); + + foreach (var move in legal) + { + int index = ChessMoveEncoding.Encode(move); + Assert.InRange(index, 0, ChessMoveEncoding.Size - 1); + Assert.True(seen.Add(index), $"index {index} collided (move {move.From}->{move.To} {move.Promotion})"); + + // Applying the ENCODED index must land on the same position as making the move directly. + string viaIndex = ChessFen.ToFen(game.Apply(state, index)); + string direct = ChessFen.ToFen(ChessRules.MakeMove(state, move)); + Assert.Equal(direct, viaIndex); + } + + // The game exposes exactly one index per legal move (no collisions collapsed the set). + Assert.Equal(legal.Count, game.LegalMoves(state).Count); + } + + [Fact] + public void Result_detects_checkmate_stalemate_and_ongoing() + { + var game = new ChessGame(); + Assert.Equal(GameResult.Ongoing, game.Result(ChessState.StartPosition())); + // Fool's mate: Black has delivered mate; White (to move) is checkmated → Loss for the side to move. + Assert.Equal(GameResult.Loss, game.Result(ChessFen.Parse("rnb1kbnr/pppp1ppp/8/4p3/6Pq/5P2/PPPPP2P/RNBQKBNR w KQkq - 1 3"))); + // Classic stalemate: Black to move, no legal move, not in check → Draw. + Assert.Equal(GameResult.Draw, game.Result(ChessFen.Parse("7k/5Q2/6K1/8/8/8/8/8 b - - 0 1"))); + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessPerftTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessPerftTests.cs new file mode 100644 index 0000000..bdd2bc9 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessPerftTests.cs @@ -0,0 +1,59 @@ +using MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// +/// Perft (performance test): count the legal-move tree to a fixed depth and match the published node counts for +/// standard positions. This is the non-negotiable correctness gate for the chess move generator (PLAN M39.2) — +/// it exercises castling, en passant, promotion, pins, and check evasion in exactly the ways hand-written tests +/// miss. Shallow depths run in the fast bucket; the multi-million-node depths are marked Slow. +/// +public class ChessPerftTests +{ + private const string StartPos = ChessFen.StartFen; + private const string Kiwipete = "r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1"; + private const string Position3 = "8/2p5/3p4/KP5r/1R3p1k/8/4P1P1/8 w - - 0 1"; + private const string Position4 = "r3k2r/Pppp1ppp/1b3nbN/nP6/BBP1P3/q4N2/Pp1P2PP/R2Q1RK1 w kq - 0 1"; + private const string Position5 = "rnbq1k1r/pp1Pbppp/2p5/8/2B5/8/PPP1NnPP/RNBQK2R w KQ - 1 8"; + private const string Position6 = "r4rk1/1pp1qppp/p1np1n2/2b1p1B1/2B1P1b1/P1NP1N2/1PP1QPPP/R4RK1 w - - 0 10"; + + [Theory] + // Startpos — the baseline. + [InlineData(StartPos, 1, 20L)] + [InlineData(StartPos, 2, 400L)] + [InlineData(StartPos, 3, 8902L)] + // Kiwipete — dense with castling, pins and captures. + [InlineData(Kiwipete, 1, 48L)] + [InlineData(Kiwipete, 2, 2039L)] + [InlineData(Kiwipete, 3, 97862L)] + // Position 3 — en passant and rook endgame checks. + [InlineData(Position3, 1, 14L)] + [InlineData(Position3, 2, 191L)] + [InlineData(Position3, 3, 2812L)] + // Position 4 — promotions and discovered checks. + [InlineData(Position4, 1, 6L)] + [InlineData(Position4, 2, 264L)] + [InlineData(Position4, 3, 9467L)] + // Position 5 — castling rights and promotions. + [InlineData(Position5, 1, 44L)] + [InlineData(Position5, 2, 1486L)] + [InlineData(Position5, 3, 62379L)] + // Position 6 — a quiet middlegame. + [InlineData(Position6, 1, 46L)] + [InlineData(Position6, 2, 2079L)] + [InlineData(Position6, 3, 89890L)] + public void Perft_matches_published_counts(string fen, int depth, long expected) + => Assert.Equal(expected, ChessRules.Perft(ChessFen.Parse(fen), depth)); + + [Theory] + [Trait("Category", "Slow")] + [InlineData(StartPos, 4, 197281L)] + [InlineData(StartPos, 5, 4865609L)] + [InlineData(Kiwipete, 4, 4085603L)] + [InlineData(Position3, 4, 43238L)] + [InlineData(Position3, 5, 674624L)] + [InlineData(Position4, 4, 422333L)] + [InlineData(Position5, 4, 2103487L)] + public void Perft_deep_matches_published_counts(string fen, int depth, long expected) + => Assert.Equal(expected, ChessRules.Perft(ChessFen.Parse(fen), depth)); +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs new file mode 100644 index 0000000..1c0e334 --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs @@ -0,0 +1,33 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// `--game chess` entry point (PLAN M39.2): AlphaZero-style self-play on chess — the second consumer of the reusable +/// self-play stack ( + ), which is reused UNCHANGED; only the +/// perft-verified is new. CPU-only and honestly bounded (a small MLP over a flattened board → +/// legal, steadily-improving play, not engine strength). Flags: --sims, --games, --eval-games, --hidden, plus the +/// common --hours/--data/--seed/--lr. +/// +internal static class ChessLab +{ + public static void Run(string[] args) + { + var a = new CliArgs(args); + double hours = a.Dbl("--hours", 1); + string dataDir = a.Str("--data", "data"); + ulong seed = a.ULong("--seed", 1); + float learningRate = a.Flt("--lr", 1e-3f); + int hidden = a.Int("--hidden", 256); // the net trunk is [hidden, hidden] + int sims = a.Int("--sims", 64); // modest — chess movegen per node is heavy on CPU + int gamesPerChunk = a.Int("--games", 8); + int evalGames = a.Int("--eval-games", 10); + bool evalOnly = a.Has("--eval-only"); + + var game = new ChessGame(); + var cfg = new Mcts.Config(Simulations: sims); + LabHost.Run(args, dataDir, hours, evalOnly, useGpu: false, + _ => new SelfPlayCampaign(game, "chess", seed, learningRate, hidden, cfg, gamesPerChunk, + tempMoves: 12, evalGames: evalGames, maxPlies: 200), + CampaignCli.ConsoleAndCsv(Path.Combine(dataDir, "logs", "chess-selfplay.csv"))); + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs index 2962ad5..65d2579 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs @@ -26,6 +26,7 @@ if (game.Equals("snake", StringComparison.OrdinalIgnoreCase)) { SnakeLab.Run(args); return; } if (game.Equals("fruitcake", StringComparison.OrdinalIgnoreCase)) { FruitCakeLab.Run(args); return; } if (game.Equals("connect4", StringComparison.OrdinalIgnoreCase)) { Connect4Lab.Run(args); return; } + if (game.Equals("chess", StringComparison.OrdinalIgnoreCase)) { ChessLab.Run(args); return; } } } From 0ccb177ed77fbe03d6746092054c565b99ccf316 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 22:03:42 +0200 Subject: [PATCH 04/33] M39.3 (robustness slice): opponent-diversity self-play MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The direct code answer to "two self-trained AIs co-adapt into a narrow style, so a novel/weak human move disorients the net and it loses to a beginner" (the exploitability failure — real even at superhuman level, cf. the KataGo cyclic-group exploit). SelfPlayCampaign gains --opponent-random : that fraction of games are learner(MCTS)-vs-random-legal instead of pure self-play, so the net trains on the off-distribution positions a blunder/unexpected move reaches and learns to punish them. Only the learner's positions are recorded, with a constant learner-perspective outcome z (distinct from self-play's alternating z). Default 0 → behaviour unchanged. Wired into --game connect4 and --game chess. (Primary robustness is still search-from-the-actual-position; Dirichlet noise + temperature already broaden self-play. League/opening-diversity/adversarial fine-tune remain M39.3 future work — see the PRD robustness note. NoisyNets is NOT the tool here — it perturbs the policy, not position coverage.) Behaviour-preserving at the default; 356 fast tests green + the new learner-vs-random contract test (frac 1.0). PLAN M39 marked M39.1+M39.2 shipped. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/PLAN.md | 14 +++- .../SelfPlayCampaignTests.cs | 27 ++++++++ .../ChessLab.cs | 3 +- .../Connect4Lab.cs | 4 +- .../SelfPlayCampaign.cs | 67 +++++++++++++++++-- 5 files changed, 105 insertions(+), 10 deletions(-) diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 6813f8c..a4ba929 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1359,7 +1359,19 @@ shallow/leaky, PR #27's correct call), the 3 imitation/policy campaigns *as whol policy campaign SHA256-bitwise-identical vs baseline; `--viz` live for every game; net **negative** line diff with every shared module carrying an interface comment. See `BOILERPLATE_REDUCTION_PRD.md`. -## M39 — Self-play training (Connect-4 → chess) *(planned 2026-07-12; see `CHESS_SELFPLAY_PRD.md`)* 🔜 +## M39 — Self-play training (Connect-4 → chess) *(2026-07-12; branch `m39-chess-selfplay-plan`, PR #32; see `CHESS_SELFPLAY_PRD.md`)* — M39.1 + M39.2 SHIPPED + +**Status (2026-07-12):** **M39.1 (rails on Connect-4) and M39.2 (chess) SHIPPED**, each its own commit on +`m39-chess-selfplay-plan`. The reusable stack — `IZeroSumGame` + `Core/Planning/Mcts.cs` (PUCT) + +`PolicyValueTraining` + `SelfPlayCampaign` — is in Core/Lab; Connect-4 and chess are both consumers, the +latter reusing the rails **unchanged**. Chess movegen is **perft-verified** (25/25 published counts, incl. startpos +depth 5 and Kiwipete depth 4); the 4672 move encoding round-trips (encode→decode→apply) with no collisions; a +`--game chess` run from random init plays legal chess and beats a random-legal opponent. A **robustness slice of +M39.3** also shipped: `--opponent-random` mixes in learner-vs-random games so the net trains on the off-distribution +positions an unexpected move reaches (the direct code answer to "a novel move disorients the AI"). 361 fast tests +green (perft, MCTS-vs-negamax, encoding round-trip, both self-play contracts). The rest of M39.3 (batched-leaf MCTS, +GPU/conv, league play, web page) remains optional future work. Honest scope holds: legal, steadily-improving play, +not engine strength. **Problem.** The SDK trains via reward (DQN/PPO), a forward model (DAVI), or an exact oracle (cube/Rush Hour imitation). Chess has no cheap oracle and a reactive net plateaus — the repo's own history says **search is the lever**. The missing paradigm is diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs index 3c57662..e1fa029 100644 --- a/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/SelfPlayCampaignTests.cs @@ -54,4 +54,31 @@ public void SelfPlay_Plays_Checkpoints_AndResumesTheNet() dir.Delete(recursive: true); } } + + /// The M39.3 robustness path: with opponentRandomFrac = 1 every game is learner-vs-random, which + /// exercises the separate value-credit assignment (constant learner-perspective z, not the alternating self-play + /// z). Asserts it plays, records samples, trains, and evaluates without error. + [Fact] + [Trait("Category", "Slow")] + public void SelfPlay_against_a_random_opponent_trains_and_evaluates() + { + var dir = Directory.CreateTempSubdirectory("connect4-selfplay-random"); + try + { + var store = new FileModelStore(dir.FullName); + var c = new Lab::SelfPlayCampaign(new Connect4Game(), "connect4", seed: 2, + learningRate: 1e-3f, hidden: 32, selfPlayCfg: new Mcts.Config(Simulations: 8), + gamesPerChunk: 6, tempMoves: 2, evalGames: 2, windowCapacity: 4000, maxPlies: 64, + targetGames: 0, opponentRandomFrac: 1.0); + + Assert.False(c.Resume(store)); + Assert.Equal(6, c.TrainChunk()); + Assert.Contains(c.Evaluate().Metrics, m => m.Name == "winRate"); + c.Dispose(); + } + finally + { + dir.Delete(recursive: true); + } + } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs index 1c0e334..a584289 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs @@ -21,13 +21,14 @@ public static void Run(string[] args) int sims = a.Int("--sims", 64); // modest — chess movegen per node is heavy on CPU int gamesPerChunk = a.Int("--games", 8); int evalGames = a.Int("--eval-games", 10); + double opponentRandom = a.Dbl("--opponent-random", 0); // fraction of games vs a random opponent (robustness) bool evalOnly = a.Has("--eval-only"); var game = new ChessGame(); var cfg = new Mcts.Config(Simulations: sims); LabHost.Run(args, dataDir, hours, evalOnly, useGpu: false, _ => new SelfPlayCampaign(game, "chess", seed, learningRate, hidden, cfg, gamesPerChunk, - tempMoves: 12, evalGames: evalGames, maxPlies: 200), + tempMoves: 12, evalGames: evalGames, maxPlies: 200, opponentRandomFrac: opponentRandom), CampaignCli.ConsoleAndCsv(Path.Combine(dataDir, "logs", "chess-selfplay.csv"))); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs index 4b640b4..f6ce259 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Connect4Lab.cs @@ -20,12 +20,14 @@ public static void Run(string[] args) int sims = a.Int("--sims", 100); // MCTS simulations per move int gamesPerChunk = a.Int("--games", 32); int evalGames = a.Int("--eval-games", 20); + double opponentRandom = a.Dbl("--opponent-random", 0); // fraction of games vs a random opponent (robustness) bool evalOnly = a.Has("--eval-only"); var game = new Connect4Game(); var cfg = new Mcts.Config(Simulations: sims); LabHost.Run(args, dataDir, hours, evalOnly, useGpu: false, - _ => new SelfPlayCampaign(game, "connect4", seed, learningRate, hidden, cfg, gamesPerChunk, evalGames: evalGames), + _ => new SelfPlayCampaign(game, "connect4", seed, learningRate, hidden, cfg, gamesPerChunk, + evalGames: evalGames, opponentRandomFrac: opponentRandom), CampaignCli.ConsoleAndCsv(Path.Combine(dataDir, "logs", "connect4-selfplay.csv"))); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs index 7cf8039..3ad1529 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SelfPlayCampaign.cs @@ -28,6 +28,7 @@ internal sealed class SelfPlayCampaign : ITrainingCampaign, INetworkTele private readonly int[] _hidden; private readonly int _gamesPerChunk, _tempMoves, _evalGames, _windowCapacity, _maxPlies; private readonly long _targetGames; + private readonly double _opponentRandomFrac; // fraction of self-play games where the learner faces a random opponent private readonly Mcts.Config _selfPlayCfg, _evalCfg; private readonly SeedSequence _seeds; @@ -42,8 +43,9 @@ internal sealed class SelfPlayCampaign : ITrainingCampaign, INetworkTele public SelfPlayCampaign(IZeroSumGame game, string environmentId, ulong seed, float learningRate, int hidden, Mcts.Config selfPlayCfg, int gamesPerChunk = 32, int tempMoves = 8, int evalGames = 20, - int windowCapacity = 40_000, int maxPlies = 512, long targetGames = 0) + int windowCapacity = 40_000, int maxPlies = 512, long targetGames = 0, double opponentRandomFrac = 0) { + _opponentRandomFrac = opponentRandomFrac; _game = game; _environmentId = environmentId; _learningRate = learningRate; @@ -149,6 +151,20 @@ public void Dispose() { } // ── Self-play ──────────────────────────────────────────────────────────────────────────────────────────────── private void PlayGame() + { + // A fraction of games pit the learner (MCTS) against a RANDOM opponent, so the net trains on the + // off-distribution positions a weak/unexpected move reaches — the direct fix for "a novel move disorients + // the AI" (PLAN M39.3). The rest are pure self-play. Opening variety otherwise comes from temperature. + if (_opponentRandomFrac > 0 && _searchRng.NextDouble() < _opponentRandomFrac) + PlayVsRandom(learnerFirst: (_totalGames & 1) == 0); + else + PlaySelfPlay(); + _totalGames++; + } + + // Both sides are the current net + MCTS; every position is a training sample. The outcome z alternates sign each + // ply (zero-sum): the terminal result is for the side to move there, so the LAST position played gets its negation. + private void PlaySelfPlay() { var state = _game.Root(); var obsHistory = new List(64); @@ -166,8 +182,6 @@ private void PlayGame() ply++; } - // Assign the outcome z back through the game, alternating sign each ply (zero-sum): the terminal result is - // for the side to move in the terminal state, so the LAST position played (opposite mover) gets its negation. float zTerminalMover = _game.Result(state) switch { GameResult.Loss => -1f, GameResult.Win => 1f, _ => 0f }; float z = -zTerminalMover; for (int i = obsHistory.Count - 1; i >= 0; i--) @@ -175,7 +189,46 @@ private void PlayGame() AddSample(new Sample(obsHistory[i], piHistory[i], z)); z = -z; } - _totalGames++; + } + + // The learner (net + MCTS) plays one colour and records its positions; the opponent plays random-legal moves. The + // learner's colour is constant, so every recorded position takes the same outcome z (from the learner's view). + private void PlayVsRandom(bool learnerFirst) + { + var state = _game.Root(); + var obsHistory = new List(64); + var piHistory = new List(64); + + bool learnerToMove = learnerFirst; + int ply = 0; + GameResult result; + while ((result = _game.Result(state)) == GameResult.Ongoing && ply < _maxPlies) + { + int move; + if (learnerToMove) + { + float[] pi = Mcts.Search(_game, state, Evaluate, _selfPlayCfg, _searchRng); + var obs = new float[_game.ObservationSize]; + _game.WriteObservation(state, obs); + obsHistory.Add(obs); + piHistory.Add(pi); + move = SelectMove(pi, ply); + } + else move = RandomLegalMove(state, _searchRng); + state = _game.Apply(state, move); + learnerToMove = !learnerToMove; + ply++; + } + + // learnerToMove now = the side to move in the terminal state. Result is for that side. + float z = result switch + { + GameResult.Loss => learnerToMove ? -1f : 1f, // side-to-move lost → learner lost iff it was to move + GameResult.Win => learnerToMove ? 1f : -1f, + _ => 0f, // draw or ply-cap + }; + for (int i = 0; i < obsHistory.Count; i++) + AddSample(new Sample(obsHistory[i], piHistory[i], z)); } private void AddSample(Sample sample) @@ -238,7 +291,7 @@ private double ArenaVsRandom() GameResult result; while ((result = _game.Result(state)) == GameResult.Ongoing && ply < _maxPlies) { - int move = mover == modelSide ? ModelMove(state) : RandomLegalMove(state); + int move = mover == modelSide ? ModelMove(state) : RandomLegalMove(state, _evalRng); state = _game.Apply(state, move); mover = 3 - mover; ply++; @@ -258,10 +311,10 @@ private int ModelMove(TState state) return best; } - private int RandomLegalMove(TState state) + private int RandomLegalMove(TState state, Xoshiro256StarStar rng) { var legal = _game.LegalMoves(state); - return legal[_evalRng.NextInt(legal.Count)]; + return legal[rng.NextInt(legal.Count)]; } // ── Live telemetry (INetworkTelemetrySource): read-only snapshot of the current net ── From 820c56e15dcd04ffb4f30e96a1da81936ad50891 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 22:40:21 +0200 Subject: [PATCH 05/33] =?UTF-8?q?M39:=20chess=20--demo=20=E2=80=94=20play/?= =?UTF-8?q?print=20one=20AI=20game=20(net+MCTS=20vs=20random)=20as=20FENs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A small dev helper to WATCH the self-taught net play: `--game chess --demo` loads the trained checkpoint and plays the net (White, MCTS) vs a random-legal opponent, printing the FEN after each ply. Used to capture a game for a browser replay; not part of training. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../ChessDemo.cs | 81 +++++++++++++++++++ .../ChessLab.cs | 3 + 2 files changed, 84 insertions(+) create mode 100644 tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessDemo.cs diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessDemo.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessDemo.cs new file mode 100644 index 0000000..e77e747 --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessDemo.cs @@ -0,0 +1,81 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Planning; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +/// +/// `--game chess --demo`: play one game with the self-taught net (net + MCTS, most-visited move) as White against a +/// random-legal opponent, printing the FEN after every ply — a watchable, decisive game that shows the AI exploiting +/// the opponent's blunders. Loads the trained checkpoint from the data dir if present, else plays from a fresh net. +/// Not part of training. +/// +internal static class ChessDemo +{ + public static void Run(string dataDir, int sims, ulong seed, int maxPlies) + { + var game = new ChessGame(); + var store = new FileModelStore(dataDir); + PolicyValueNet net; + using (var s = store.TryOpenRead("chess", "az")) + net = s is not null + ? PolicyValueNet.Load(s, "selfplay-pv", game.ObservationSize, game.PolicySize) + : new PolicyValueNet(game.ObservationSize, [256, 256], game.PolicySize, new Xoshiro256StarStar(seed)); + Console.Error.WriteLine(store.TryOpenRead("chess", "az") is null ? "(fresh net)" : "(loaded trained net)"); + + var rng = new Xoshiro256StarStar(seed); + var cfg = new Mcts.Config(Simulations: sims, RootNoiseFrac: 0f); + + (float[] Priors, float Value) Evaluate(ChessState st) + { + var obs = new float[game.ObservationSize]; + game.WriteObservation(st, obs); + using (GradMode.NoGrad()) + { + var (logits, value) = net.Forward(new Tensor(obs, 1, obs.Length)); + var legal = game.LegalMoves(st); + var priors = new float[game.PolicySize]; + float max = float.NegativeInfinity; + foreach (int m in legal) if (logits.Data[m] > max) max = logits.Data[m]; + float sum = 0f; + foreach (int m in legal) { float e = MathF.Exp(logits.Data[m] - max); priors[m] = e; sum += e; } + if (sum > 0f) foreach (int m in legal) priors[m] /= sum; + return (priors, MathF.Tanh(value.Data[0])); + } + } + + const bool aiIsWhite = true; // the AI plays White; a random-legal mover plays Black + var state = game.Root(); + Console.WriteLine(ChessFen.ToFen(state)); // starting position first + int ply = 0; + while (game.Result(state) == GameResult.Ongoing && ply < maxPlies) + { + int move; + if (state.WhiteToMove == aiIsWhite) + { + var pi = Mcts.Search(game, state, Evaluate, cfg, rng); + move = 0; + for (int a = 1; a < pi.Length; a++) if (pi[a] > pi[move]) move = a; + } + else + { + var legal = game.LegalMoves(state); + move = legal[rng.NextInt(legal.Count)]; + } + state = game.Apply(state, move); + Console.WriteLine(ChessFen.ToFen(state)); + ply++; + } + // Report from White's (the AI's) perspective. + var final = game.Result(state); + string outcome = final switch + { + GameResult.Loss => state.WhiteToMove ? "AI (White) lost" : "AI (White) won", + GameResult.Win => state.WhiteToMove ? "AI (White) won" : "AI (White) lost", + GameResult.Draw => "draw", + _ => "unfinished (ply cap)", + }; + Console.Error.WriteLine($"result: {outcome} after {ply} plies ({sims} sims/move)"); + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs index a584289..6c36dbe 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/ChessLab.cs @@ -24,6 +24,9 @@ public static void Run(string[] args) double opponentRandom = a.Dbl("--opponent-random", 0); // fraction of games vs a random opponent (robustness) bool evalOnly = a.Has("--eval-only"); + // --demo: play one self-play game with the (trained) net and print FENs to watch — no training. + if (a.Has("--demo")) { ChessDemo.Run(dataDir, sims, seed, a.Int("--demo-plies", 100)); return; } + var game = new ChessGame(); var cfg = new Mcts.Config(Simulations: sims); LabHost.Run(args, dataDir, hours, evalOnly, useGpu: false, From d2c3e6ac24c7b1961e4d951fedb8dfa5117a4a61 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 22:56:55 +0200 Subject: [PATCH 06/33] =?UTF-8?q?docs(M40):=20plan=20=E2=80=94=20play=20th?= =?UTF-8?q?e=20chess=20AI=20in=20the=20browser=20via=20Polyglot=20single-s?= =?UTF-8?q?ource?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The FruitCake pattern for chess: write the inference path once in a .pg, transpile to C# (training) + TypeScript (browser); the browser downloads and parses the .ckpt and runs net+MCTS client-side — zero server inference. Feasibility VERIFIED by transpiling a probe: Polyglot supports Math.exp/tanh/ log/sqrt + bitwise ops (transcendentals aren't bit-exact C#/JS, but chess inference doesn't need that). Adds docs/prd/CHESS_WEB_POLYGLOT_PRD.md (incl. a §8 reference appendix: files to port, the exact .ckpt byte-format for the TS parser, store ids, commands) + an M40 milestone in PLAN.md. Phased: M40.1 single-source the engine (re-validate perft) → M40.2 net-forward + MCTS in the .pg + a TS ckpt parser → M40.3 Angular chess page (client-side play). Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/CHESS_WEB_POLYGLOT_PRD.md | 136 +++++++++++++++++++++++++++++ docs/prd/PLAN.md | 41 +++++++++ 2 files changed, 177 insertions(+) create mode 100644 docs/prd/CHESS_WEB_POLYGLOT_PRD.md diff --git a/docs/prd/CHESS_WEB_POLYGLOT_PRD.md b/docs/prd/CHESS_WEB_POLYGLOT_PRD.md new file mode 100644 index 0000000..6664ba0 --- /dev/null +++ b/docs/prd/CHESS_WEB_POLYGLOT_PRD.md @@ -0,0 +1,136 @@ +# Play the chess AI in the browser — single-source via MintPlayer.Polyglot — PRD + +**Status:** Planned · 2026-07-12 · branch TBD (off the M39 branch / `master`) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M40 · **Depends on:** M39 (the chess engine, the self-play-trained `PolicyValueNet`, the `.ckpt` format) and the MintPlayer.Polyglot single-source toolchain already used by FruitCake ([../ARCHITECTURE.md](../ARCHITECTURE.md) §10; `Environments/FruitCake/polyglot/`). + +## 1. Problem + +You can *watch* a game the chess AI played (a replay), but you can't *play against it in the browser* — there's no web chess page, and the AI is C# (net + MCTS), so a static page can't run it. The wrong fix is a server `ChessController` that runs MCTS per move (server inference, per-viewer CPU — the exact thing M32/M33 removed for FruitCake/Snake). The **right** fix is the pattern MintPlayer.Polyglot exists for and FruitCake already proves: **write the inference path once in a `.pg`, transpile to C# (training/serving) and TypeScript (browser), and run the whole AI client-side, with the browser downloading and parsing the `.ckpt`.** Zero server inference. + +## 2. Feasibility — verified + +The open question was whether chess's inference math and move-gen fit Polyglot's subset. Probed by transpiling (2026-07-12): + +- **`Math.exp`, `Math.tanh`, `Math.log`, `Math.sqrt` all transpile** (with `import { Math } from "std.math"`) — so MCTS's masked-softmax priors, the `tanh` value head, and the PUCT `sqrt` term are all expressible. (FruitCake avoided transcendentals because its physics must be *bit-exact* across C#/JS; only `+ - * / sqrt` are. **Chess inference does not need bit-exactness** — the browser AI just needs to play well, not match C# to the ULP — so non-bit-exact `exp`/`tanh` are fine here.) +- **Bitwise operators `& | << >>` transpile** — castling-rights flags and any bit tricks work (booleans are also an option). +- **Polyglot itself is extendable if a gap appears.** The language/toolchain lives at `C:\Repos\MintPlayer.Polyglot`; a missing feature (e.g. a `match`/`switch`, an enum, a Math function) can be added there and PR'd. The known limits to design around (from the FruitCake solver): **no nested-generic params** (`List>` is mishandled — use flat `List` + offsets, as `PgDuelingNet` does), and prefer `i32` constants + `if/else` over enums/`switch` unless we add them upstream. + +**Conclusion:** the chess engine + observation + net forward + MCTS can be single-sourced to `chess_solver.pg` and run client-side. This is a build, not a research risk. + +## 3. Goal & success criteria + +- **Play a full game against the self-taught AI in the browser** — an interactive board (you're White; click/drag to move, legal moves highlighted), the AI (Black) replies from a `.pg`-single-sourced net + MCTS running **in your browser**, over the shipped `.ckpt` it downloads. No server move computation. +- **Single source.** The chess **engine** (legal move-gen, apply, terminal, move encoding), the **observation**, the **net forward**, and the **MCTS** live once in `chess_solver.pg`, transpiled to C# (compiled into the Environments assembly, like FruitCake) and to committed TypeScript for the web client. A parity test pins the C# facade to the generated core; **perft still passes 25/25 against the generated engine** (the M39.2 gate carries over onto the single source). +- **Zero server inference / load-only web app.** The web app only serves the `.ckpt` (static file, `application/octet-stream` mapping like the FruitCake net) and the SPA. Per-viewer server cost is zero — consistent with M32/M33. +- **Honest scope.** It's the M39 network — a small MLP, briefly CPU-trained — so it plays *legal, still-learning* chess, beatable by a decent human. Browser MCTS is tuned (modest sims) for interactive latency (~1–2 s/move). +- **Training unchanged.** Training stays on the SDK (autograd/GEMM); the `.pg` is **inference + rules** only. The engine (generated C#) is shared by training's self-play; the net *forward* in the `.pg` is inference-only (training learns the weights via autograd, exactly as FruitCake). + +**Non-goals.** Engine strength; a server chess endpoint; bit-exact C#/JS transcendentals (not needed — this isn't a shared fairness sim); underpromotion-free or reduced rules (the engine is already full + perft-verified); training in the browser. + +## 4. Design — follow the FruitCake pattern + +| Piece | Where | Note | +|---|---|---| +| `chess_solver.pg` | `Environments/Chess/polyglot/` | **single source:** the engine (a `Pg`-prefixed board + `legalMoves`/`apply`/`result`/`isAttacked`), `writeObservation`, `PgPolicyValueNet.forward` (flat-array trunk + policy/value heads, `tanh` value), the 4672 `encode`/`decode`, and a chess `mcts` (PUCT, masked-softmax priors, `sqrt` exploration, **no Dirichlet** — inference only). Transpiled to C# (`obj/`, build-time) + committed `chess_solver.ts`. | +| C# facade | `Environments/Chess/ChessGame.cs` (rewired) | Wraps the generated `Pg` engine as `IZeroSumGame` (float/host view), so M39's `SelfPlayCampaign`/`ChessGame` keep working on the single source. Replaces the hand-written `ChessBoard.cs` engine internals; **perft tests re-point at the generated core**. | +| TS `.ckpt` parser | `ClientApp/src/app/chess/chess-net.ts` | Mirrors the C# `PolicyValueNet` reader (magic `RLNC`, kind `selfplay-pv`, v2 trunk-widths, params in `Parameters()` order) → builds the generated `PgPolicyValueNet`. The one non-Polyglot piece (binary I/O), exactly like `fruitcake-net.ts`. | +| Chess director | `ClientApp/src/app/chess/chess-director.ts` | Runs the transpiled `mcts` + net over the loaded `.ckpt` to pick the AI's reply — the analog of `fruit-cake-director.ts`. | +| Angular page | `ClientApp/src/app/chess/` + route/nav | Interactive board (unicode glyphs or simple SVG), human move input validated by the transpiled `legalMoves`, AI reply from the director, status (check/mate/draw), last-move highlight. | +| Shipped weights | `wwwroot/models/chess.az.ckpt` (LFS) + the `.ckpt` MIME mapping | Fetched by the browser; the checkpoint the director loads. | + +## 5. Phased plan (each ends on a green build + its gate) + +- **M40.1 — single-source the engine.** Port `ChessBoard`/`ChessRules` + `ChessMoveEncoding` into `chess_solver.pg`; wire the Polyglot MSBuild transpile; rewire `ChessGame`/facade onto the generated core; **re-run perft (25/25) + the encoding round-trip on the generated engine.** De-risks the biggest port (rules) first; training self-play still passes its contract test. +- **M40.2 — single-source the inference math + TS parser.** Add `PgPolicyValueNet.forward` (flat arrays, `tanh`), `writeObservation`, and a chess `mcts` to the `.pg`; a `chess-net.ts` `.ckpt` parser; a parity test (C# `PolicyValueNet.Forward` vs the generated `PgPolicyValueNet` on a fixed position, within f32 tolerance). Emit the committed `chess_solver.ts`. +- **M40.3 — the browser page.** `chess-director.ts` + the Angular chess component (board, input, AI reply, status) + route/nav; ship `wwwroot/models/chess.az.ckpt`; the `.ckpt` static-file mapping. **Gate:** play a full legal game vs the AI in the browser with no server move calls (verified via the network panel / Playwright); tune sims for latency. A longer training run first, so it's a worthier opponent. + +## 6. Risks + +1. **Browser MCTS performance.** Chess move-gen per node is heavier than FruitCake's depth-3 search; hundreds of sims/move in transpiled TS may be slow. Mitigation: modest inference sims (100–300), a ply/time budget, and profile — it's a latency knob, not a correctness one. (Batched-leaf / WASM are later levers.) +2. **Polyglot language gaps.** The port may want a `switch`/`match`, enums, or a Math function not yet in `std.math`. Mitigation: design around them (i32 consts + `if/else`, flat arrays — proven by FruitCake) **or add them to `C:\Repos\MintPlayer.Polyglot` and PR** (now an available option). No nested-generic params. +3. **Re-validating the engine after the port.** Rewriting a perft-verified engine in another language risks a subtle regression. Mitigation: perft (25/25) + the encoding round-trip run against the *generated* engine as the M40.1 gate; a facade-vs-core parity test. +4. **Honest strength.** Still a small, briefly-trained net → beatable. Framed as a from-scratch, self-taught, learning AI (the charm), not an engine. + +## 7. Verification + +- **M40.1:** solution builds (the `.pg` transpiles into the assembly); **perft 25/25** + encoding round-trip on the generated engine; `SelfPlayCampaign` chess contract test still green. +- **M40.2:** C#-vs-generated `PgPolicyValueNet` parity within f32 tolerance on fixed positions; `chess_solver.ts` emitted + committed; the TS parser round-trips a shipped `.ckpt` (a small Node/Jest or a Playwright check). +- **M40.3:** in-browser, a full legal game vs the AI, **no `/api/chess/*` move calls** (Network panel/Playwright), check/mate/draw detected, last-move highlight. Honest latency (~1–2 s/move at the chosen sims). + +See [PLAN.md](PLAN.md) M40. + +## 8. Reference appendix (execution details — read this if picking up cold) + +### 8a. What M39 already shipped (branch `m39-chess-selfplay-plan`, PR #32, stacked on the M38 branch #31) +The chess self-play stack is **done and committed**; M40 builds on it. Key artifacts: +- **Reusable seam/search/training (Core + Lab):** `Core/Planning/IZeroSumGame.cs` (+ `GameResult`), `Core/Planning/Mcts.cs` + (PUCT: select `Q+U` → expand-with-priors → net-leaf → **value negated every ply** → `Search` returns the root + visit-count π; Dirichlet root noise + `Config(Simulations, Cpuct=1.25, DirichletAlpha=0.3, RootNoiseFrac=0.25)`), + `Lab/PolicyValueTraining.TrainStep` (soft-CE(π) + MSE(`tanh` value, z)), `Lab/SelfPlayCampaign` + (`ITrainingCampaign`; plays MCTS self-play → `(obs,π,z)` window → trains `PolicyValueNet`; `--opponent-random` frac + mixes learner-vs-random games; store ids below). +- **Chess (Environments/Chess/):** `ChessBoard.cs` (`ChessState` + `ChessRules`: `LegalMoves`, `MakeMove`, + `IsSquareAttacked`, `InCheck`, `Result`, `Perft`; mailbox `sbyte[64]`, sq = rank*8+file, White moves +rank; piece + codes ±1..6 = P,N,B,R,Q,K; castling byte bits WK=1,WQ=2,BK=4,BQ=8; **threefold repetition NOT modelled** — 50-move + + ply cap bound loops), `ChessFen.cs`, `ChessMoveEncoding.cs` (AlphaZero **4672 = 64×73**: 56 queen planes [8 dir ×7 + dist] + 8 knight + 9 underpromotion; queen-promo rides the queen planes, decoded promo inferred on apply), + `ChessGame.cs` (`IZeroSumGame`; `PolicySize=4672`, `ObservationSize=1152` = 18 planes×64: [0–5] White + P,N,B,R,Q,K, [6–11] Black, [12] side-to-move, [13–16] castling WK/WQ/BK/BQ, [17] en-passant square). +- **Lab:** `--game chess` (`ChessLab.cs`, flags `--sims --games --eval-games --hidden --opponent-random`), + `--game chess --demo` (`ChessDemo.cs`: net as White + MCTS vs random Black, prints one FEN per ply to stdout), + `--game connect4` (`Connect4Lab.cs` + `Environments/Connect4/`). +- **Net:** the shared `Core/Nn/PolicyValueNet.cs` (variable-depth ReLU trunk → policy logits + scalar value), used + directly (no chess wrapper). **Value head is linear** — self-play wraps it in `tanh` at the call site. +- **Tests:** `ChessPerftTests` (25/25, incl. startpos d5=4,865,609, Kiwipete d4=4,085,603), `ChessEncodingTests` + (encode→decode→apply round-trip + mate/stalemate), `Connect4Tests` (MCTS-vs-negamax), `SelfPlayCampaignTests` + (resume roundtrip + vs-random). 355 fast + deep-perft (Slow). + +### 8b. Model-store ids / checkpoint (what the browser must parse) +`SelfPlayCampaign` saves to store `(env, algo)`: net = `("chess","az")` written `PolicyValueNet.Save(stream, kind:"selfplay-pv")`; +Adam = `("chess","az-adam")`. File on disk: `/chess.az.ckpt`. **The TS `.ckpt` parser (chess-net.ts) reads +(little-endian, mirroring `CheckpointFormat` + `PolicyValueNet.Save`; see `fruitcake-net.ts` for the exact idiom):** +``` +uint32 magic = 0x434E4C52 ("RLNC") +string kind = "selfplay-pv" (BinaryWriter 7-bit-encoded-int length prefix + UTF-8 bytes) +int32 version (= 2) +int32 trunkCount, then trunkCount × int32 (the hidden widths, e.g. 256,256) +per layer, in Parameters() order = [each trunk layer, then policyHead, then valueHead]: + int32 wCount + wCount × float32 (weight, row-major [inDim, outDim] → w[i*outDim + o]) + int32 bCount + bCount × float32 (bias) +``` +`inputSize` (1152) and `actions` (4672) are **not** stored — supply them from `ChessGame` (they’re Load params). +Forward: `x = obs; for each trunk layer: x = relu(x·W + b); logits = x·W_pol + b_pol; value = tanh(x·W_val + b_val)`. +MCTS priors = masked-softmax of `logits` over legal moves; leaf value = `value`. + +### 8c. Commands +``` +# train chess (writes /chess.az.ckpt; time-bounded; --opponent-random for robustness): +dotnet run -c Release --project tools/MintPlayer.AI.ReinforcementLearning.Lab -- \ + --game chess --hours 2 --sims 48 --games 10 --opponent-random 0.25 --data --seed 7 +# watch a game (prints FENs; loads the checkpoint from ): +dotnet run -c Release --project tools/…Lab -- --game chess --demo --sims 400 --demo-plies 120 --data --seed 11 +# perft/encoding gate: dotnet test --filter "FullyQualifiedName~ChessPerftTests" (add Category!=Slow for shallow) +# Polyglot transpile (bundled CLI; win-x64): +# ~/.nuget/packages/mintplayer.polyglot.msbuild/0.3.1/tools/win-x64/polyglot.exe \ +# build .pg --target typescript --out (and --target csharp) +``` + +### 8d. Polyglot facts (from the FruitCake precedent — `Environments/FruitCake/polyglot/`) +- Source: `fruitcake_solver.pg` (single source) → C# in `obj/` (build-time, MSBuild PackageReference + `MintPlayer.Polyglot.MSBuild` v0.3.1, `PrivateAssets=all`) + committed TS `fruitcake_solver.ts`. macOS dev must + point `$(PolyglotTool)` at a local `polyglot` binary. `dotnet watch` re-transpiles on save. +- Syntax: `import { List } from "std.collections"`, `import { Math } from "std.math"`; `record X(f: t)`; `class X { var f: t; init(...) {...}; fn m(...): t {...} }`; `fn f(a: i32): t => expr`; `for i in 0..n`, `for v in list`; + types `f64 i32 bool List T?`; `Math.min/max/sqrt/exp/tanh/log/PI`. Generated C# types are `Pg`-prefixed + internal, + wrapped by a hand-written **facade** (`FruitCakeWorld.cs`); a **parity test** pins facade↔core. +- **Constraints:** no nested-generic params (`List>` mishandled → flat `List` + offsets, see + `PgDuelingNet`); prefer `i32` consts + `if/else` over enums/`switch`. If blocked, extend Polyglot at + `C:\Repos\MintPlayer.Polyglot` and PR (owner-authorized 2026-07-12). +- Browser wiring (analogs to build): `chess-net.ts` (`.ckpt` parser → `PgPolicyValueNet`), `chess-director.ts` + (runs transpiled mcts+net), the Angular `chess/` component + route; ship `wwwroot/models/chess.az.ckpt` (LFS) + the + `.ckpt`→`application/octet-stream` static-file mapping in `Program.cs` (see how FruitCake's net is served). + +### 8e. Status of the "watch" demo (already delivered) +A browser **replay** artifact of one AI game exists (published via the Artifact tool from `--game chess --demo` +FENs). That is watch-only; M40 is the interactive *play-against-it* page. The demo net was trained ~21 min on CPU +(reached ~50% vs random, policy loss 4.4→2.2) — legal but weak; longer training recommended before M40.3. diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index a4ba929..f8116a3 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1416,6 +1416,47 @@ chess is the headline consumer. **Left out of v1:** DQN/PPO for self-play (wrong window instead), bitboards (unless a bench forces it), superhuman strength. **Whole-milestone gate:** builds 0/0; fast suite green; perft passes; both games' self-play win-rate rises vs a fixed baseline; resume roundtrips. See `CHESS_SELFPLAY_PRD.md`. +## M40 — Play the chess AI in the browser (single-source via MintPlayer.Polyglot) *(planned 2026-07-12; see `CHESS_WEB_POLYGLOT_PRD.md`)* 🔜 + +**Goal.** Play the self-taught chess AI (M39) **in the browser**, client-side, with **zero server inference** — the +FruitCake pattern (ARCHITECTURE §10): write the inference path once in a `.pg`, transpile to C# (training/serving) + +TypeScript (browser); the browser downloads and parses the `.ckpt` and runs the net + MCTS locally. The wrong path +is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); the right path is single-source. + +**Feasibility — VERIFIED (2026-07-12)** by transpiling a probe with the bundled `polyglot.exe`: +- `Math.exp`, `Math.tanh`, `Math.log`, `Math.sqrt` all transpile (need `import { Math } from "std.math"`), so MCTS's + masked-softmax priors + `tanh` value + PUCT `sqrt` are expressible. Transcendentals are **not** bit-exact across + C#/JS (only `+ - * / sqrt` are — the reason FruitCake avoided them) but chess **inference doesn't need bit-exactness** + (the browser AI must play well, not match C# to the ULP), so `exp`/`tanh` are fine. +- Bitwise ops `& | << >>` transpile (castling flags; booleans also fine). +- **Polyglot itself is extendable:** the toolchain lives at `C:\Repos\MintPlayer.Polyglot` — a missing feature + (`switch`/enum/Math fn) can be **added there and PR'd** (owner-authorized). Known limits to design around + (from the FruitCake solver): **no nested-generic params** (use flat `List` + offsets like `PgDuelingNet`), + prefer `i32` consts + `if/else` over enums/`switch`. + +**Phases (each ends on a green build + its gate).** +- **M40.1 — single-source the engine.** Port `ChessBoard`/`ChessRules` + `ChessMoveEncoding` into + `Environments/Chess/polyglot/chess_solver.pg`; the `MintPlayer.Polyglot.MSBuild` PackageReference already in the + Environments `.csproj` (v0.3.1) transpiles every `**/*.pg` to `obj/` before CoreCompile. Rewire `ChessGame`/a facade + onto the generated `Pg` core (replacing the hand-written engine internals). **Gate: perft 25/25 + the encoding + round-trip on the GENERATED engine** (the M39.2 gate carries onto the single source); the chess `SelfPlayCampaign` + contract test stays green. +- **M40.2 — single-source the inference math + a TS `.ckpt` parser.** Add `PgPolicyValueNet.forward` (flat-array + trunk + policy/value heads, `tanh` value), `writeObservation`, and a chess `mcts` (PUCT, masked-softmax, `sqrt`, + **no Dirichlet** — inference only) to the `.pg`. Add `ClientApp/src/app/chess/chess-net.ts` (mirrors the C# + `PolicyValueNet` reader — see the PRD appendix for the exact bytes). **Gate:** C#-vs-generated parity within f32 + tol on a fixed position; committed `chess_solver.ts` emitted. +- **M40.3 — the browser page.** `chess-director.ts` (runs the transpiled `mcts`+net over the loaded `.ckpt`, like + `fruit-cake-director.ts`) + an Angular chess component (board, click-to-move validated by the transpiled + `legalMoves`, AI reply, check/mate/draw, last-move highlight) + route/nav; ship `wwwroot/models/chess.az.ckpt` + (LFS) + the `.ckpt` MIME mapping in `Program.cs`. **Gate:** a full legal game vs the AI in-browser with **no + `/api/chess/*` move calls** (Network panel/Playwright); tune inference sims (~100–300) for ~1–2 s/move latency. + Train a longer run first for a worthier opponent. + +**Honest scope:** the M39 net is a small, briefly-CPU-trained MLP → legal, still-learning chess, beatable by a decent +human. **Non-goals:** engine strength, a server chess endpoint, bit-exact transcendentals, browser training. See +`CHESS_WEB_POLYGLOT_PRD.md` (incl. its reference appendix: files to port, checkpoint byte-format, commands). + ## Testing strategy (cross-cutting, from research) 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket. From 7c228c427f6c095657c4d73d445a676842db7849 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 23:18:57 +0200 Subject: [PATCH 07/33] =?UTF-8?q?M40.1=20=E2=80=94=20single-source=20the?= =?UTF-8?q?=20chess=20engine=20via=20Polyglot=20(perft=2025/25=20on=20gene?= =?UTF-8?q?rated=20core)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Port the perft-verified chess engine (ChessBoard: ChessState+ChessRules, and ChessMoveEncoding) into one Polyglot source, chess_solver.pg, transpiled to C# (obj/, build-time) — the same source the browser client will run in M40.2/M40.3. - chess_solver.pg: internal Pg-prefixed core (PgChessState + PgChessMove). Board is List[64], promotion/castling are i32, perft returns i32 (fits every tested depth; widened to long in C#). Rays are bounded for+continue (no while); delta tables are List<(i32,i32)> iterated with tuple destructuring — every construct is one proven by fruitcake_solver.pg. - ChessState / ChessRules / ChessMoveEncoding are now thin C# facades over the generated core; PieceType, ChessMove, and all public signatures are unchanged, so ChessGame, ChessFen, SelfPlayCampaign, and both chess test files are untouched. - ChessGame's seam (LegalMoves/Apply/Result) delegates to the core's legalMoveIndices/applyIndex/result, single-sourcing the index logic training and the browser both use. Perft recurses entirely in-core (no per-node marshalling). Gate: perft 25/25 on the GENERATED engine (incl. startpos d5 = 4,865,609, Kiwipete d4 = 4,085,603, ~10s), encoding round-trip + terminal detection green, SelfPlay chess contract green, full fast suite 355/355 (no FruitCake/Snake/MountainCar regression from the shared re-transpile). PLAN M40.1 marked shipped. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/PLAN.md | 19 +- .../Chess/ChessBoard.cs | 362 ++----------- .../Chess/ChessGame.cs | 33 +- .../Chess/ChessMoveEncoding.cs | 79 +-- .../Chess/polyglot/chess_solver.pg | 501 ++++++++++++++++++ 5 files changed, 581 insertions(+), 413 deletions(-) create mode 100644 src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index f8116a3..1ef6365 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1435,12 +1435,19 @@ is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); th prefer `i32` consts + `if/else` over enums/`switch`. **Phases (each ends on a green build + its gate).** -- **M40.1 — single-source the engine.** Port `ChessBoard`/`ChessRules` + `ChessMoveEncoding` into - `Environments/Chess/polyglot/chess_solver.pg`; the `MintPlayer.Polyglot.MSBuild` PackageReference already in the - Environments `.csproj` (v0.3.1) transpiles every `**/*.pg` to `obj/` before CoreCompile. Rewire `ChessGame`/a facade - onto the generated `Pg` core (replacing the hand-written engine internals). **Gate: perft 25/25 + the encoding - round-trip on the GENERATED engine** (the M39.2 gate carries onto the single source); the chess `SelfPlayCampaign` - contract test stays green. +- **M40.1 — single-source the engine. ✅ SHIPPED 2026-07-12.** Ported `ChessBoard`/`ChessRules` + `ChessMoveEncoding` + into `Environments/Chess/polyglot/chess_solver.pg` (internal `Pg`-prefixed core: `PgChessState` + `PgChessMove` — + board `List[64]`, promotion/castling as `i32`, bounded `for`+`continue` rays, `List<(i32,i32)>` delta tables; + every construct proven by `fruitcake_solver.pg`). The `MintPlayer.Polyglot.MSBuild` PackageReference (v0.3.1) already + transpiles every `**/*.pg` to `obj/` before CoreCompile — 0.3.1 already bundles win-x64 + linux-x64 + linux-arm64, + so CI (ubuntu) is unaffected. `ChessState`/`ChessRules`/`ChessMoveEncoding` are now **thin C# facades** over the core + (public API + both test files unchanged); `ChessGame`'s seam (`LegalMoves`/`Apply`/`Result`) delegates to the core's + `legalMoveIndices`/`applyIndex`/`result` so training and the browser share one implementation; `perft` recurses + entirely in-core. **Gate MET: perft 25/25 on the generated engine** (incl. startpos d5 = 4,865,609, Kiwipete + d4 = 4,085,603, ~10 s), encoding round-trip + terminal-detection green, `SelfPlayCampaign` chess contract green, and + the full fast suite 355/355 (no FruitCake/Snake/MountainCar regression from the shared re-transpile). Note: the + multi-`.pg` stale-prelude codegen bug (CS0101/CS0260) reappears on incremental add of a new `.pg` — cleared per the + CLAUDE.md fix (`rm obj/*/net10.0/polyglot/*.cs`) then rebuilt clean. - **M40.2 — single-source the inference math + a TS `.ckpt` parser.** Add `PgPolicyValueNet.forward` (flat-array trunk + policy/value heads, `tanh` value), `writeObservation`, and a chess `mcts` (PUCT, masked-softmax, `sqrt`, **no Dirichlet** — inference only) to the `.pg`. Add `ClientApp/src/app/chess/chess-net.ts` (mirrors the C# diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs index ea1e37b..3f899fd 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessBoard.cs @@ -9,342 +9,88 @@ public enum PieceType : byte { None = 0, Pawn = 1, Knight = 2, Bishop = 3, Rook public readonly record struct ChessMove(byte From, byte To, PieceType Promotion = PieceType.None); /// -/// A chess position (mailbox board + side to move + castling rights + en-passant target + halfmove clock) and the -/// full rules: legal move generation (castling, en passant, promotion, pins/checks), make-move, attack detection, -/// and terminal detection (checkmate, stalemate, 50-move, insufficient material). Correctness-first (clone-per-move, -/// no bitboards); verified by perft. Squares are 0..63 with file = sq & 7, rank = sq >> 3; White moves -/// toward higher ranks. Board cells: 0 = empty, +1..+6 = White P,N,B,R,Q,K, −1..−6 = Black (sign = colour). -/// Threefold-repetition is intentionally not modelled in v1 (see PLAN M39); the 50-move rule and the self-play +/// A chess position — the public, ergonomic view (mailbox , side to move, castling rights, +/// en-passant target, halfmove clock). This is a thin facade over the single-source engine transpiled from +/// polyglot/chess_solver.pg (PgChessState) — the same source the browser client runs. The board is +/// immutable: every move produces a fresh state. Squares are 0..63 (file = sq % 8, rank = sq / 8); White moves +/// toward higher ranks. Cells: 0 = empty, +1..+6 = White P,N,B,R,Q,K, −1..−6 = Black (sign = colour). +/// Threefold-repetition is intentionally not modelled (see PLAN M39); the 50-move rule and the self-play /// ply cap bound looping games. /// public sealed class ChessState { - public sbyte[] Squares { get; } // length 64 - public bool WhiteToMove { get; } - public byte Castling { get; } // bit 0 = White O-O, 1 = White O-O-O, 2 = Black O-O, 3 = Black O-O-O - public sbyte EnPassant { get; } // target square a pawn could capture onto, or -1 - public byte HalfmoveClock { get; } + internal readonly PgChessState Core; + private sbyte[]? _squares; public const byte CastleWK = 1, CastleWQ = 2, CastleBK = 4, CastleBQ = 8; - public ChessState(sbyte[] squares, bool whiteToMove, byte castling, sbyte enPassant, byte halfmoveClock) - { - Squares = squares; - WhiteToMove = whiteToMove; - Castling = castling; - EnPassant = enPassant; - HalfmoveClock = halfmoveClock; - } - - public static ChessState StartPosition() => ChessFen.Parse(ChessFen.StartFen); -} - -/// Static chess rules over . -public static class ChessRules -{ - private static int File(int sq) => sq & 7; - private static int Rank(int sq) => sq >> 3; - private static PieceType Type(sbyte piece) => (PieceType)Math.Abs(piece); - private static bool IsWhite(sbyte piece) => piece > 0; - - private static readonly (int df, int dr)[] KnightDeltas = - [(1, 2), (2, 1), (2, -1), (1, -2), (-1, -2), (-2, -1), (-2, 1), (-1, 2)]; - private static readonly (int df, int dr)[] KingDeltas = - [(1, 0), (1, 1), (0, 1), (-1, 1), (-1, 0), (-1, -1), (0, -1), (1, -1)]; - private static readonly (int df, int dr)[] BishopDirs = [(1, 1), (-1, 1), (1, -1), (-1, -1)]; - private static readonly (int df, int dr)[] RookDirs = [(1, 0), (-1, 0), (0, 1), (0, -1)]; - - // ── Attack detection ──────────────────────────────────────────────────────────────────────────────────────── - - /// Whether is attacked by the given colour's pieces. - public static bool IsSquareAttacked(ChessState s, int square, bool byWhite) - { - int tf = File(square), tr = Rank(square); - var b = s.Squares; - - // Pawns: a pawn on the attacking side attacks diagonally toward its forward direction. - int pawnDir = byWhite ? 1 : -1; - sbyte pawn = (sbyte)(byWhite ? 1 : -1); - foreach (int df in stackalloc[] { -1, 1 }) - { - int f = tf + df, r = tr - pawnDir; // a pawn on (r) attacking (tr) sits one rank behind in its dir - if ((uint)f < 8 && (uint)r < 8 && b[r * 8 + f] == pawn) return true; - } - - foreach (var (df, dr) in KnightDeltas) - { - int f = tf + df, r = tr + dr; - if ((uint)f < 8 && (uint)r < 8 && b[r * 8 + f] == (sbyte)(byWhite ? 2 : -2)) return true; - } - - foreach (var (df, dr) in KingDeltas) - { - int f = tf + df, r = tr + dr; - if ((uint)f < 8 && (uint)r < 8 && b[r * 8 + f] == (sbyte)(byWhite ? 6 : -6)) return true; - } - - // Sliders: bishops/queens on diagonals, rooks/queens on ranks/files. - if (RayHits(b, tf, tr, BishopDirs, byWhite, PieceType.Bishop)) return true; - if (RayHits(b, tf, tr, RookDirs, byWhite, PieceType.Rook)) return true; - return false; - } - - private static bool RayHits(sbyte[] b, int tf, int tr, (int df, int dr)[] dirs, bool byWhite, PieceType slider) - { - foreach (var (df, dr) in dirs) - { - int f = tf + df, r = tr + dr; - while ((uint)f < 8 && (uint)r < 8) - { - sbyte piece = b[r * 8 + f]; - if (piece != 0) - { - if (IsWhite(piece) == byWhite) - { - var t = Type(piece); - if (t == slider || t == PieceType.Queen) return true; - } - break; // blocked - } - f += df; r += dr; - } - } - return false; - } - - public static int KingSquare(ChessState s, bool white) - { - sbyte king = (sbyte)(white ? 6 : -6); - for (int sq = 0; sq < 64; sq++) if (s.Squares[sq] == king) return sq; - return -1; - } - - public static bool InCheck(ChessState s, bool white) - => IsSquareAttacked(s, KingSquare(s, white), byWhite: !white); - - // ── Move generation ───────────────────────────────────────────────────────────────────────────────────────── - - /// All fully-legal moves for the side to move (own king not left in check). - public static List LegalMoves(ChessState s) - { - var pseudo = PseudoLegal(s); - var legal = new List(pseudo.Count); - foreach (var move in pseudo) - { - var next = MakeMove(s, move); - if (!InCheck(next, white: s.WhiteToMove)) legal.Add(move); // the side that just moved must be safe - } - return legal; - } - - private static List PseudoLegal(ChessState s) - { - var moves = new List(48); - bool white = s.WhiteToMove; - var b = s.Squares; - - for (int sq = 0; sq < 64; sq++) - { - sbyte piece = b[sq]; - if (piece == 0 || IsWhite(piece) != white) continue; - int f = File(sq), r = Rank(sq); - switch (Type(piece)) - { - case PieceType.Pawn: PawnMoves(s, sq, f, r, white, moves); break; - case PieceType.Knight: StepMoves(b, sq, f, r, white, KnightDeltas, moves); break; - case PieceType.King: StepMoves(b, sq, f, r, white, KingDeltas, moves); CastleMoves(s, white, moves); break; - case PieceType.Bishop: SlideMoves(b, sq, f, r, white, BishopDirs, moves); break; - case PieceType.Rook: SlideMoves(b, sq, f, r, white, RookDirs, moves); break; - case PieceType.Queen: - SlideMoves(b, sq, f, r, white, BishopDirs, moves); - SlideMoves(b, sq, f, r, white, RookDirs, moves); - break; - } - } - return moves; - } + internal ChessState(PgChessState core) => Core = core; - private static void StepMoves(sbyte[] b, int sq, int f, int r, bool white, (int df, int dr)[] deltas, List moves) + public ChessState(sbyte[] squares, bool whiteToMove, byte castling, sbyte enPassant, byte halfmoveClock) { - foreach (var (df, dr) in deltas) - { - int nf = f + df, nr = r + dr; - if ((uint)nf >= 8 || (uint)nr >= 8) continue; - int to = nr * 8 + nf; - sbyte target = b[to]; - if (target == 0 || IsWhite(target) != white) moves.Add(new ChessMove((byte)sq, (byte)to)); - } + var cells = new List(64); + for (int i = 0; i < 64; i++) cells.Add(squares[i]); + Core = new PgChessState(cells, whiteToMove, castling, enPassant, halfmoveClock); } - private static void SlideMoves(sbyte[] b, int sq, int f, int r, bool white, (int df, int dr)[] dirs, List moves) + /// The 64-cell mailbox (a cached snapshot of the core board — read-only). + public sbyte[] Squares { - foreach (var (df, dr) in dirs) + get { - int nf = f + df, nr = r + dr; - while ((uint)nf < 8 && (uint)nr < 8) + if (_squares is null) { - int to = nr * 8 + nf; - sbyte target = b[to]; - if (target == 0) moves.Add(new ChessMove((byte)sq, (byte)to)); - else { if (IsWhite(target) != white) moves.Add(new ChessMove((byte)sq, (byte)to)); break; } - nf += df; nr += dr; + _squares = new sbyte[64]; + for (int i = 0; i < 64; i++) _squares[i] = (sbyte)Core.squares[i]; } + return _squares; } } - private static void PawnMoves(ChessState s, int sq, int f, int r, bool white, List moves) - { - var b = s.Squares; - int dir = white ? 1 : -1; - int startRank = white ? 1 : 6; - int lastRank = white ? 7 : 0; - - // Single / double push. - int one = (r + dir) * 8 + f; - if (b[one] == 0) - { - AddPawn(moves, sq, one, r + dir == lastRank); - if (r == startRank) - { - int two = (r + 2 * dir) * 8 + f; - if (b[two] == 0) moves.Add(new ChessMove((byte)sq, (byte)two)); - } - } - - // Captures (incl. en passant). - foreach (int df in stackalloc[] { -1, 1 }) - { - int nf = f + df, nr = r + dir; - if ((uint)nf >= 8 || (uint)nr >= 8) continue; - int to = nr * 8 + nf; - sbyte target = b[to]; - if (target != 0 && IsWhite(target) != white) AddPawn(moves, sq, to, nr == lastRank); - else if (to == s.EnPassant) moves.Add(new ChessMove((byte)sq, (byte)to)); // en passant onto the empty ep square - } - } + public bool WhiteToMove => Core.whiteToMove; + public byte Castling => (byte)Core.castling; // bit 0 = White O-O, 1 = White O-O-O, 2 = Black O-O, 3 = Black O-O-O + public sbyte EnPassant => (sbyte)Core.enPassant; + public byte HalfmoveClock => (byte)Core.halfmoveClock; - private static void AddPawn(List moves, int from, int to, bool promotion) - { - if (promotion) - { - moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Queen)); - moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Rook)); - moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Bishop)); - moves.Add(new ChessMove((byte)from, (byte)to, PieceType.Knight)); - } - else moves.Add(new ChessMove((byte)from, (byte)to)); - } + public static ChessState StartPosition() => ChessFen.Parse(ChessFen.StartFen); - private static void CastleMoves(ChessState s, bool white, List moves) - { - var b = s.Squares; - int rank = white ? 0 : 7; - int kingSq = rank * 8 + 4; - if (b[kingSq] != (sbyte)(white ? 6 : -6)) return; - if (IsSquareAttacked(s, kingSq, byWhite: !white)) return; // can't castle out of check + // Move ↔ generated-move mapping (promotion piece codes coincide with PieceType's byte values, so the cast is + // a no-op renaming). Shared by ChessRules and ChessMoveEncoding — the two facades over the single source. + internal static PgChessMove ToPg(ChessMove m) => new(m.From, m.To, (int)m.Promotion); + internal static ChessMove FromPg(PgChessMove m) => new((byte)m.from, (byte)m.to, (PieceType)m.promotion); +} - byte kSide = white ? ChessState.CastleWK : ChessState.CastleBK; - byte qSide = white ? ChessState.CastleWQ : ChessState.CastleBQ; +/// +/// The chess rules — a public facade over the single-source engine (chess_solver.pg → PgChessState): +/// legal move generation (castling, en passant, promotion, pins/checks), make-move, attack detection, and terminal +/// detection (checkmate, stalemate, 50-move, insufficient material). Correctness is pinned by perft, which +/// recurses entirely inside the generated core (no per-node marshalling). +/// +public static class ChessRules +{ + public static bool IsSquareAttacked(ChessState s, int square, bool byWhite) => s.Core.isSquareAttacked(square, byWhite); - // King-side: squares f,g empty; king doesn't pass through/into attack. - if ((s.Castling & kSide) != 0 && b[rank * 8 + 5] == 0 && b[rank * 8 + 6] == 0 - && !IsSquareAttacked(s, rank * 8 + 5, !white) && !IsSquareAttacked(s, rank * 8 + 6, !white)) - moves.Add(new ChessMove((byte)kingSq, (byte)(rank * 8 + 6))); + public static int KingSquare(ChessState s, bool white) => s.Core.kingSquare(white); - // Queen-side: squares b,c,d empty; king passes over d,c (b need not be safe, only empty). - if ((s.Castling & qSide) != 0 && b[rank * 8 + 1] == 0 && b[rank * 8 + 2] == 0 && b[rank * 8 + 3] == 0 - && !IsSquareAttacked(s, rank * 8 + 3, !white) && !IsSquareAttacked(s, rank * 8 + 2, !white)) - moves.Add(new ChessMove((byte)kingSq, (byte)(rank * 8 + 2))); - } + public static bool InCheck(ChessState s, bool white) => s.Core.inCheck(white); - // ── Make move ─────────────────────────────────────────────────────────────────────────────────────────────── - - /// The position after (a fresh state; is unchanged). - /// Castling / en passant / promotion / double-push are inferred from the board. - public static ChessState MakeMove(ChessState s, ChessMove move) + /// All fully-legal moves for the side to move (own king not left in check). + public static List LegalMoves(ChessState s) { - var b = (sbyte[])s.Squares.Clone(); - bool white = s.WhiteToMove; - sbyte piece = b[move.From]; - var type = Type(piece); - bool capture = b[move.To] != 0; - sbyte newEp = -1; - byte castling = s.Castling; - - b[move.From] = 0; - - // En passant capture: the taken pawn sits beside the destination, not on it. - if (type == PieceType.Pawn && move.To == s.EnPassant && s.EnPassant >= 0) - { - int capturedSq = move.To + (white ? -8 : 8); - b[capturedSq] = 0; - capture = true; - } - - // Place the piece (promotion swaps in the chosen piece). - b[move.To] = move.Promotion != PieceType.None - ? (sbyte)((white ? 1 : -1) * (int)move.Promotion) - : piece; - - // Double push sets the en-passant target square. - if (type == PieceType.Pawn && Math.Abs(Rank(move.To) - Rank(move.From)) == 2) - newEp = (sbyte)((move.From + move.To) / 2); - - // Castling: move the rook too. - if (type == PieceType.King && Math.Abs(File(move.To) - File(move.From)) == 2) - { - int rank = Rank(move.From); - if (File(move.To) == 6) { b[rank * 8 + 5] = b[rank * 8 + 7]; b[rank * 8 + 7] = 0; } // king-side - else { b[rank * 8 + 3] = b[rank * 8 + 0]; b[rank * 8 + 0] = 0; } // queen-side - } - - // Update castling rights: king move clears both; rook move/capture clears that corner. - if (type == PieceType.King) castling &= (byte)(white ? ~(ChessState.CastleWK | ChessState.CastleWQ) : ~(ChessState.CastleBK | ChessState.CastleBQ)); - castling &= CastlingMask(move.From); - castling &= CastlingMask(move.To); // a rook captured on its home square loses that right too - - byte halfmove = (byte)(type == PieceType.Pawn || capture ? 0 : s.HalfmoveClock + 1); - return new ChessState(b, !white, castling, newEp, halfmove); + var core = s.Core.legalMoves(); + var moves = new List(core.Count); + foreach (var m in core) moves.Add(ChessState.FromPg(m)); + return moves; } - // Clears the castling right whose rook/king home square is 'sq' (identity mask otherwise). - private static byte CastlingMask(int sq) => sq switch - { - 0 => unchecked((byte)~ChessState.CastleWQ), // a1 rook - 7 => unchecked((byte)~ChessState.CastleWK), // h1 rook - 56 => unchecked((byte)~ChessState.CastleBQ), // a8 rook - 63 => unchecked((byte)~ChessState.CastleBK), // h8 rook - _ => 0xFF, - }; - - // ── Terminal detection ────────────────────────────────────────────────────────────────────────────────────── + /// The position after (a fresh state; is unchanged). + public static ChessState MakeMove(ChessState s, ChessMove move) => new(s.Core.makeMove(ChessState.ToPg(move))); - public static bool IsFiftyMove(ChessState s) => s.HalfmoveClock >= 100; + public static bool IsFiftyMove(ChessState s) => s.Core.isFiftyMove(); - public static bool IsInsufficientMaterial(ChessState s) - { - int knights = 0, bishops = 0; - foreach (sbyte p in s.Squares) - { - switch (Type(p)) - { - case PieceType.None or PieceType.King: break; - case PieceType.Knight: knights++; break; - case PieceType.Bishop: bishops++; break; - default: return false; // a pawn/rook/queen = enough material - } - } - return knights + bishops <= 1; // K vs K, K+N vs K, K+B vs K (not exhaustive, but the common draws) - } + public static bool IsInsufficientMaterial(ChessState s) => s.Core.isInsufficientMaterial(); - /// Perft: the number of leaf nodes at — the movegen correctness oracle. - public static long Perft(ChessState s, int depth) - { - if (depth == 0) return 1; - var moves = LegalMoves(s); - if (depth == 1) return moves.Count; - long nodes = 0; - foreach (var move in moves) nodes += Perft(MakeMove(s, move), depth - 1); - return nodes; - } + /// Perft: the number of leaf nodes at — the movegen correctness oracle. The + /// count is produced entirely inside the generated engine (i32 there; widened to long here). + public static long Perft(ChessState s, int depth) => s.Core.perft(depth); } diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs index 58ef92c..d0593a2 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessGame.cs @@ -19,33 +19,18 @@ public sealed class ChessGame : IZeroSumGame public ChessState Root(ulong? seed = null) => ChessState.StartPosition(); - public IReadOnlyList LegalMoves(ChessState state) - { - var moves = ChessRules.LegalMoves(state); - var indices = new int[moves.Count]; - for (int i = 0; i < moves.Count; i++) indices[i] = ChessMoveEncoding.Encode(moves[i]); - return indices; - } + // The seam methods delegate to the single-source core so training and the browser client share one + // implementation of "legal indices / apply an index / terminal result". + public IReadOnlyList LegalMoves(ChessState state) => state.Core.legalMoveIndices(); - public ChessState Apply(ChessState state, int move) - { - var decoded = ChessMoveEncoding.Decode(move); - // A queen-promotion rides the queen planes and decodes as None → promote the pawn to a Queen by default. - if (decoded.Promotion == PieceType.None - && (PieceType)Math.Abs(state.Squares[decoded.From]) == PieceType.Pawn - && (decoded.To >> 3) is 0 or 7) - decoded = decoded with { Promotion = PieceType.Queen }; - return ChessRules.MakeMove(state, decoded); - } + public ChessState Apply(ChessState state, int move) => new(state.Core.applyIndex(move)); - public GameResult Result(ChessState state) + public GameResult Result(ChessState state) => state.Core.result() switch { - if (ChessRules.LegalMoves(state).Count == 0) - return ChessRules.InCheck(state, state.WhiteToMove) ? GameResult.Loss : GameResult.Draw; // mate vs stalemate - if (ChessRules.IsFiftyMove(state) || ChessRules.IsInsufficientMaterial(state)) - return GameResult.Draw; - return GameResult.Ongoing; - } + 1 => GameResult.Loss, // side to move is checkmated + 2 => GameResult.Draw, // stalemate / 50-move / insufficient material + _ => GameResult.Ongoing, + }; public void WriteObservation(ChessState state, Span destination) { diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs index f7ddea4..5de0116 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/ChessMoveEncoding.cs @@ -7,90 +7,19 @@ namespace MintPlayer.AI.ReinforcementLearning.Environments.Chess; /// {capture-left, straight, capture-right}). Distinct legal moves always map to distinct indices, so the search can /// tell them apart; queen-promotions ride the queen planes (decoded promotion is inferred as Queen when a pawn /// reaches the last rank — see ). +/// A thin facade over the single-source encoder (chess_solver.pg → PgChessState.encode/decode), +/// so the browser client and the training seam share one implementation. /// public static class ChessMoveEncoding { public const int PlanesPerSquare = 73; public const int Size = 64 * PlanesPerSquare; // 4672 - // 8 sliding directions (df, dr), indices 0..7 — shared by encode and decode. - private static readonly (int df, int dr)[] Dirs = - [(0, 1), (1, 1), (1, 0), (1, -1), (0, -1), (-1, -1), (-1, 0), (-1, 1)]; - // 8 knight deltas (df, dr), indices 0..7. - private static readonly (int df, int dr)[] Knights = - [(1, 2), (2, 1), (2, -1), (1, -2), (-1, -2), (-2, -1), (-2, 1), (-1, 2)]; - - private static int File(int sq) => sq & 7; - private static int Rank(int sq) => sq >> 3; - /// The action index (0..4671) for a legal move. Queen-promotions use the queen planes; N/B/R /// promotions use the underpromotion planes. - public static int Encode(ChessMove move) - { - int df = File(move.To) - File(move.From); - int dr = Rank(move.To) - Rank(move.From); - int plane; - - if (move.Promotion is PieceType.Knight or PieceType.Bishop or PieceType.Rook) - { - int pieceIdx = move.Promotion switch { PieceType.Knight => 0, PieceType.Bishop => 1, _ => 2 }; - plane = 64 + pieceIdx * 3 + (df + 1); // df ∈ {-1,0,1} → {capture-left, straight, capture-right} - } - else if (IsKnight(df, dr)) - { - plane = 56 + KnightIndex(df, dr); - } - else - { - int dir = DirIndex(Math.Sign(df), Math.Sign(dr)); - int dist = Math.Max(Math.Abs(df), Math.Abs(dr)); - plane = dir * 7 + (dist - 1); - } - return move.From * PlanesPerSquare + plane; - } + public static int Encode(ChessMove move) => PgChessState.encode(ChessState.ToPg(move)); /// Decodes an action index to a move. A queen-promotion decodes with — the /// caller promotes a pawn reaching the last rank to a Queen by default. - public static ChessMove Decode(int index) - { - int from = index / PlanesPerSquare, plane = index % PlanesPerSquare; - int f = File(from), r = Rank(from); - - if (plane >= 64) - { - int p = plane - 64; - var promo = (p / 3) switch { 0 => PieceType.Knight, 1 => PieceType.Bishop, _ => PieceType.Rook }; - int df = (p % 3) - 1; - int dr = r == 6 ? 1 : -1; // a pawn on the 7th rank promotes upward, on the 2nd rank downward - return new ChessMove((byte)from, (byte)((r + dr) * 8 + (f + df)), promo); - } - if (plane >= 56) - { - var (df, dr) = Knights[plane - 56]; - return new ChessMove((byte)from, (byte)((r + dr) * 8 + (f + df))); - } - { - var (df, dr) = Dirs[plane / 7]; - int dist = plane % 7 + 1; - return new ChessMove((byte)from, (byte)((r + dr * dist) * 8 + (f + df * dist))); - } - } - - private static bool IsKnight(int df, int dr) - { - int a = Math.Abs(df), b = Math.Abs(dr); - return (a == 1 && b == 2) || (a == 2 && b == 1); - } - - private static int KnightIndex(int df, int dr) - { - for (int i = 0; i < Knights.Length; i++) if (Knights[i] == (df, dr)) return i; - throw new ArgumentException($"Not a knight move: ({df},{dr})."); - } - - private static int DirIndex(int sdf, int sdr) - { - for (int i = 0; i < Dirs.Length; i++) if (Dirs[i] == (sdf, sdr)) return i; - throw new ArgumentException($"Not a straight/diagonal direction: ({sdf},{sdr})."); - } + public static ChessMove Decode(int index) => ChessState.FromPg(PgChessState.decode(index)); } diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg new file mode 100644 index 0000000..bd94fae --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg @@ -0,0 +1,501 @@ +// chess_solver.pg — single-source port of the perft-verified chess engine +// (ChessBoard.cs: ChessState + ChessRules, and ChessMoveEncoding.cs). ONE source, +// transpiled to C# (obj/, build-time, wrapped by the ChessState/ChessRules facades) +// and — from M40.2 — to a committed TypeScript twin the browser runs client-side. +// +// Types are `Pg`-prefixed and stay internal; the hand-written C# facades +// (ChessBoard.cs, ChessMoveEncoding.cs) adapt this i32/List core to the public +// PieceType/ChessMove/sbyte[] API the rest of the SDK and the perft tests consume. +// Perft (the movegen correctness gate) recurses entirely in this core. +// +// Subset choices (mirror the FruitCake precedent): the mailbox board is a List +// of 64 cells (0 empty, +1..+6 White P,N,B,R,Q,K, −1..−6 Black); promotion is an i32 +// piece code (0 = none, 2..5 = N,B,R,Q); castling is an i32 bitmask (WK=1,WQ=2,BK=4, +// BQ=8). No i64 — perft returns i32 (fits every position/depth the tests probe; +// the C# facade widens to long). Ray/slide loops are bounded `for`+`continue` (no +// `while`); delta tables are List<(i32,i32)> iterated with tuple destructuring — every +// construct here is one proven by fruitcake_solver.pg. + +import { List } from "std.collections" + +// A move: from/to squares (0..63 = rank*8 + file) plus the promotion piece code +// (0 = none) for a pawn reaching the last rank. Castling / en passant / double-push +// are inferred from the board when the move is made, so they need no extra flag. +record PgChessMove(from: i32, to: i32, promotion: i32) + +class PgChessState { + const PromoKnight: i32 = 2 + const PromoBishop: i32 = 3 + const PromoRook: i32 = 4 + const PromoQueen: i32 = 5 + + const CastleWK: i32 = 1 + const CastleWQ: i32 = 2 + const CastleBK: i32 = 4 + const CastleBQ: i32 = 8 + + const Size: i32 = 4672 // 64 × 73 AlphaZero action space + const PlanesPerSquare: i32 = 73 + + var squares: List // length 64 + var whiteToMove: bool + var castling: i32 + var enPassant: i32 // target square a pawn could capture onto, or -1 + var halfmoveClock: i32 + + init(squares: List, whiteToMove: bool, castling: i32, enPassant: i32, halfmoveClock: i32) { + this.squares = squares + this.whiteToMove = whiteToMove + this.castling = castling + this.enPassant = enPassant + this.halfmoveClock = halfmoveClock + } + + // ── small helpers (i32 arithmetic std.math doesn't cover) ───────────────────────────────── + static fn fileOf(sq: i32): i32 => sq % 8 + static fn rankOf(sq: i32): i32 => sq / 8 + static fn iabs(x: i32): i32 => if x < 0 { -x } else { x } + static fn isign(x: i32): i32 => if x > 0 { 1 } else { if x < 0 { -1 } else { 0 } } + static fn imax(a: i32, b: i32): i32 => if a > b { a } else { b } + static fn onBoard(f: i32, r: i32): bool => f >= 0 && f < 8 && r >= 0 && r < 8 + static fn isWhite(piece: i32): bool => piece > 0 + static fn pieceType(piece: i32): i32 => if piece < 0 { -piece } else { piece } + // Clear the given bit(s) of a mask without a bitwise-not: drop the parts that ARE in `bits`. + static fn clearBits(mask: i32, bits: i32): i32 => mask - (mask & bits) + + // ── delta tables (built fresh; iterated with tuple destructuring) ───────────────────────── + static fn knightDeltas(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 2)); d.add((2, 1)); d.add((2, -1)); d.add((1, -2)) + d.add((-1, -2)); d.add((-2, -1)); d.add((-2, 1)); d.add((-1, 2)) + return d + } + static fn kingDeltas(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 0)); d.add((1, 1)); d.add((0, 1)); d.add((-1, 1)) + d.add((-1, 0)); d.add((-1, -1)); d.add((0, -1)); d.add((1, -1)) + return d + } + static fn bishopDirs(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 1)); d.add((-1, 1)); d.add((1, -1)); d.add((-1, -1)) + return d + } + static fn rookDirs(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 0)); d.add((-1, 0)); d.add((0, 1)); d.add((0, -1)) + return d + } + // Encoding tables (AlphaZero plane order — shared by encode and decode). + static fn encDirs(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((0, 1)); d.add((1, 1)); d.add((1, 0)); d.add((1, -1)) + d.add((0, -1)); d.add((-1, -1)); d.add((-1, 0)); d.add((-1, 1)) + return d + } + static fn encKnights(): List<(i32, i32)> { + var d: List<(i32, i32)> = List<(i32, i32)>() + d.add((1, 2)); d.add((2, 1)); d.add((2, -1)); d.add((1, -2)) + d.add((-1, -2)); d.add((-2, -1)); d.add((-2, 1)); d.add((-1, 2)) + return d + } + + fn clone(): PgChessState { + var sq: List = List() + for v in this.squares { sq.add(v) } + return PgChessState(sq, this.whiteToMove, this.castling, this.enPassant, this.halfmoveClock) + } + + // ── attack detection ────────────────────────────────────────────────────────────────────── + fn isSquareAttacked(square: i32, byWhite: bool): bool { + let tf = fileOf(square) + let tr = rankOf(square) + + // Pawns: an attacking pawn sits one rank behind the target in its forward direction. + let pawnDir = if byWhite { 1 } else { -1 } + let pawn = if byWhite { 1 } else { -1 } + for k in 0..2 { + let df = k * 2 - 1 // -1, +1 + let f = tf + df + let r = tr - pawnDir + if onBoard(f, r) && this.squares[r * 8 + f] == pawn { return true } + } + + let knight = if byWhite { 2 } else { -2 } + let kn = knightDeltas() + for (df, dr) in kn { + let f = tf + df + let r = tr + dr + if onBoard(f, r) && this.squares[r * 8 + f] == knight { return true } + } + + let king = if byWhite { 6 } else { -6 } + let kg = kingDeltas() + for (df, dr) in kg { + let f = tf + df + let r = tr + dr + if onBoard(f, r) && this.squares[r * 8 + f] == king { return true } + } + + if this.rayHits(tf, tr, bishopDirs(), byWhite, 3) { return true } // bishops/queens + if this.rayHits(tf, tr, rookDirs(), byWhite, 4) { return true } // rooks/queens + return false + } + + // A sliding piece of type `slider` (or a queen, code 5) of colour `byWhite` attacks (tf,tr) + // along one of `dirs`. Bounded to 7 steps; `blocked` stops the walk at the first obstruction. + fn rayHits(tf: i32, tr: i32, dirs: List<(i32, i32)>, byWhite: bool, slider: i32): bool { + for (df, dr) in dirs { + var found = false + var blocked = false + for step in 1..8 { + if blocked { continue } + let f = tf + df * step + let r = tr + dr * step + if !onBoard(f, r) { blocked = true; continue } + let piece = this.squares[r * 8 + f] + if piece != 0 { + if isWhite(piece) == byWhite { + let t = pieceType(piece) + if t == slider || t == 5 { found = true } + } + blocked = true + } + } + if found { return true } + } + return false + } + + fn kingSquare(white: bool): i32 { + let king = if white { 6 } else { -6 } + for sq in 0..64 { if this.squares[sq] == king { return sq } } + return -1 + } + + fn inCheck(white: bool): bool => this.isSquareAttacked(this.kingSquare(white), !white) + + // ── move generation ───────────────────────────────────────────────────────────────────── + fn legalMoves(): List { + var legal: List = List() + let pseudo = this.pseudoLegal() + for m in pseudo { + let next = this.makeMove(m) + if !next.inCheck(this.whiteToMove) { legal.add(m) } // the side that just moved must be safe + } + return legal + } + + fn pseudoLegal(): List { + var moves: List = List() + let white = this.whiteToMove + for sq in 0..64 { + let piece = this.squares[sq] + if piece == 0 || isWhite(piece) != white { continue } + let f = fileOf(sq) + let r = rankOf(sq) + let t = pieceType(piece) + if t == 1 { this.pawnMoves(sq, f, r, white, moves) } + else if t == 2 { this.stepMoves(sq, f, r, white, knightDeltas(), moves) } + else if t == 6 { this.stepMoves(sq, f, r, white, kingDeltas(), moves); this.castleMoves(white, moves) } + else if t == 3 { this.slideMoves(sq, f, r, white, bishopDirs(), moves) } + else if t == 4 { this.slideMoves(sq, f, r, white, rookDirs(), moves) } + else if t == 5 { + this.slideMoves(sq, f, r, white, bishopDirs(), moves) + this.slideMoves(sq, f, r, white, rookDirs(), moves) + } + } + return moves + } + + fn stepMoves(sq: i32, f: i32, r: i32, white: bool, deltas: List<(i32, i32)>, moves: List) { + for (df, dr) in deltas { + let nf = f + df + let nr = r + dr + if !onBoard(nf, nr) { continue } + let to = nr * 8 + nf + let target = this.squares[to] + if target == 0 || isWhite(target) != white { moves.add(PgChessMove(sq, to, 0)) } + } + } + + fn slideMoves(sq: i32, f: i32, r: i32, white: bool, dirs: List<(i32, i32)>, moves: List) { + for (df, dr) in dirs { + var blocked = false + for step in 1..8 { + if blocked { continue } + let nf = f + df * step + let nr = r + dr * step + if !onBoard(nf, nr) { blocked = true; continue } + let to = nr * 8 + nf + let target = this.squares[to] + if target == 0 { moves.add(PgChessMove(sq, to, 0)) } + else { + if isWhite(target) != white { moves.add(PgChessMove(sq, to, 0)) } + blocked = true + } + } + } + } + + fn pawnMoves(sq: i32, f: i32, r: i32, white: bool, moves: List) { + let dir = if white { 1 } else { -1 } + let startRank = if white { 1 } else { 6 } + let lastRank = if white { 7 } else { 0 } + + let one = (r + dir) * 8 + f + if this.squares[one] == 0 { + this.addPawn(sq, one, r + dir == lastRank, moves) + if r == startRank { + let two = (r + 2 * dir) * 8 + f + if this.squares[two] == 0 { moves.add(PgChessMove(sq, two, 0)) } + } + } + + for k in 0..2 { + let df = k * 2 - 1 // -1, +1 + let nf = f + df + let nr = r + dir + if !onBoard(nf, nr) { continue } + let to = nr * 8 + nf + let target = this.squares[to] + if target != 0 && isWhite(target) != white { this.addPawn(sq, to, nr == lastRank, moves) } + else if to == this.enPassant { moves.add(PgChessMove(sq, to, 0)) } // en passant onto the empty ep square + } + } + + fn addPawn(from: i32, to: i32, promotion: bool, moves: List) { + if promotion { + moves.add(PgChessMove(from, to, PromoQueen)) + moves.add(PgChessMove(from, to, PromoRook)) + moves.add(PgChessMove(from, to, PromoBishop)) + moves.add(PgChessMove(from, to, PromoKnight)) + } else { + moves.add(PgChessMove(from, to, 0)) + } + } + + fn castleMoves(white: bool, moves: List) { + let rank = if white { 0 } else { 7 } + let kingSq = rank * 8 + 4 + let ownKing = if white { 6 } else { -6 } + if this.squares[kingSq] != ownKing { return } + if this.isSquareAttacked(kingSq, !white) { return } // can't castle out of check + + let kSide = if white { CastleWK } else { CastleBK } + let qSide = if white { CastleWQ } else { CastleBQ } + + // King-side: f,g empty; king doesn't pass through/into attack. + let kEmpty = this.squares[rank * 8 + 5] == 0 && this.squares[rank * 8 + 6] == 0 + let kSafe = !this.isSquareAttacked(rank * 8 + 5, !white) && !this.isSquareAttacked(rank * 8 + 6, !white) + if (this.castling & kSide) != 0 && kEmpty && kSafe { moves.add(PgChessMove(kingSq, rank * 8 + 6, 0)) } + + // Queen-side: b,c,d empty; king passes over d,c (b need not be safe, only empty). + let qEmpty = this.squares[rank * 8 + 1] == 0 && this.squares[rank * 8 + 2] == 0 && this.squares[rank * 8 + 3] == 0 + let qSafe = !this.isSquareAttacked(rank * 8 + 3, !white) && !this.isSquareAttacked(rank * 8 + 2, !white) + if (this.castling & qSide) != 0 && qEmpty && qSafe { moves.add(PgChessMove(kingSq, rank * 8 + 2, 0)) } + } + + // ── make move ─────────────────────────────────────────────────────────────────────────── + // The position after `move` (a fresh state; this one is unchanged). Castling / en passant / + // promotion / double-push are inferred from the board. + fn makeMove(move: PgChessMove): PgChessState { + var b: List = List() + for v in this.squares { b.add(v) } + let white = this.whiteToMove + let piece = b[move.from] + let t = pieceType(piece) + var capture = b[move.to] != 0 + var newEp = -1 + var castling = this.castling + + b[move.from] = 0 + + // En passant capture: the taken pawn sits beside the destination, not on it. + if t == 1 && move.to == this.enPassant && this.enPassant >= 0 { + let capturedSq = move.to + (if white { -8 } else { 8 }) + b[capturedSq] = 0 + capture = true + } + + // Place the piece (promotion swaps in the chosen piece). + if move.promotion != 0 { + b[move.to] = (if white { 1 } else { -1 }) * move.promotion + } else { + b[move.to] = piece + } + + // Double push sets the en-passant target square. + if t == 1 && iabs(rankOf(move.to) - rankOf(move.from)) == 2 { + newEp = (move.from + move.to) / 2 + } + + // Castling: move the rook too. + if t == 6 && iabs(fileOf(move.to) - fileOf(move.from)) == 2 { + let rank = rankOf(move.from) + if fileOf(move.to) == 6 { b[rank * 8 + 5] = b[rank * 8 + 7]; b[rank * 8 + 7] = 0 } // king-side + else { b[rank * 8 + 3] = b[rank * 8 + 0]; b[rank * 8 + 0] = 0 } // queen-side + } + + // Update castling rights: a king move clears both of its sides; a rook move/capture on a + // home corner clears that right. + if t == 6 { + if white { castling = clearBits(castling, CastleWK + CastleWQ) } + else { castling = clearBits(castling, CastleBK + CastleBQ) } + } + castling = clearBits(castling, cornerRight(move.from)) + castling = clearBits(castling, cornerRight(move.to)) // a rook captured on its home square loses that right too + + let halfmove = if t == 1 || capture { 0 } else { this.halfmoveClock + 1 } + return PgChessState(b, !white, castling, newEp, halfmove) + } + + // The castling-right bit anchored at a rook/king home corner (0 = none). + static fn cornerRight(sq: i32): i32 { + if sq == 0 { return CastleWQ } // a1 rook + if sq == 7 { return CastleWK } // h1 rook + if sq == 56 { return CastleBQ } // a8 rook + if sq == 63 { return CastleBK } // h8 rook + return 0 + } + + // ── terminal detection ────────────────────────────────────────────────────────────────── + fn isFiftyMove(): bool => this.halfmoveClock >= 100 + + fn isInsufficientMaterial(): bool { + var knights = 0 + var bishops = 0 + for p in this.squares { + let t = pieceType(p) + if t == 0 || t == 6 { continue } + else if t == 2 { knights += 1 } + else if t == 3 { bishops += 1 } + else { return false } // a pawn/rook/queen = enough material + } + return knights + bishops <= 1 // K vs K, K+N vs K, K+B vs K (the common draws) + } + + // Terminal result FROM THE SIDE TO MOVE's view: 0 = ongoing, 1 = loss (checkmated), 2 = draw. + fn result(): i32 { + if this.legalMoves().count == 0 { + return if this.inCheck(this.whiteToMove) { 1 } else { 2 } // mate vs stalemate + } + if this.isFiftyMove() || this.isInsufficientMaterial() { return 2 } + return 0 + } + + // Perft: the number of leaf nodes at `depth` — the movegen correctness oracle. i32 suffices + // for every position/depth the tests probe (max ~4.9M); the C# facade widens to long. + fn perft(depth: i32): i32 { + if depth == 0 { return 1 } + let moves = this.legalMoves() + if depth == 1 { return moves.count } + var nodes = 0 + for m in moves { nodes += this.makeMove(m).perft(depth - 1) } + return nodes + } + + // ── seam helpers (index-level API the C# facade / browser consume) ──────────────────────── + fn legalMoveIndices(): List { + var out: List = List() + let moves = this.legalMoves() + for m in moves { out.add(encode(m)) } + return out + } + + // Apply an encoded action index (a queen-promotion rides the queen planes and decodes with + // promotion 0 → promote a pawn reaching the last rank to a Queen by default). + fn applyIndex(index: i32): PgChessState { + let m = decode(index) + var chosen = m + if m.promotion == 0 && pieceType(this.squares[m.from]) == 1 { + let toRank = rankOf(m.to) + if toRank == 0 || toRank == 7 { chosen = PgChessMove(m.from, m.to, PromoQueen) } + } + return this.makeMove(chosen) + } + + // ── AlphaZero 4672 move encoding (64 × 73 = from-square × plane) ────────────────────────── + // 56 queen planes (8 dir × 7 dist — also carry queen-promotions and king/pawn moves) + 8 + // knight planes + 9 underpromotion planes (N/B/R × {capture-left, straight, capture-right}). + static fn encode(move: PgChessMove): i32 { + let df = fileOf(move.to) - fileOf(move.from) + let dr = rankOf(move.to) - rankOf(move.from) + var plane = 0 + if move.promotion == PromoKnight || move.promotion == PromoBishop || move.promotion == PromoRook { + let pieceIdx = if move.promotion == PromoKnight { 0 } else { if move.promotion == PromoBishop { 1 } else { 2 } } + plane = 64 + pieceIdx * 3 + (df + 1) // df ∈ {-1,0,1} → {capture-left, straight, capture-right} + } else if isKnightMove(df, dr) { + plane = 56 + knightIndex(df, dr) + } else { + let dir = dirIndex(isign(df), isign(dr)) + let dist = imax(iabs(df), iabs(dr)) + plane = dir * 7 + (dist - 1) + } + return move.from * PlanesPerSquare + plane + } + + // Decode an action index to a move. A queen-promotion decodes with promotion 0 — the caller + // promotes a pawn reaching the last rank to a Queen by default (see applyIndex). + static fn decode(index: i32): PgChessMove { + let from = index / PlanesPerSquare + let plane = index % PlanesPerSquare + let f = fileOf(from) + let r = rankOf(from) + + if plane >= 64 { + let p = plane - 64 + let promo = if p / 3 == 0 { PromoKnight } else { if p / 3 == 1 { PromoBishop } else { PromoRook } } + let df = (p % 3) - 1 + let dr = if r == 6 { 1 } else { -1 } // a pawn on the 7th rank promotes upward, on the 2nd downward + return PgChessMove(from, (r + dr) * 8 + (f + df), promo) + } + if plane >= 56 { + let target = plane - 56 + var i = 0 + var rdf = 0 + var rdr = 0 + for (df, dr) in encKnights() { + if i == target { rdf = df; rdr = dr } + i += 1 + } + return PgChessMove(from, (r + rdr) * 8 + (f + rdf), 0) + } + let dirTarget = plane / 7 + let dist = plane % 7 + 1 + var j = 0 + var ddf = 0 + var ddr = 0 + for (df, dr) in encDirs() { + if j == dirTarget { ddf = df; ddr = dr } + j += 1 + } + return PgChessMove(from, (r + ddr * dist) * 8 + (f + ddf * dist), 0) + } + + static fn isKnightMove(df: i32, dr: i32): bool { + let a = iabs(df) + let b = iabs(dr) + return (a == 1 && b == 2) || (a == 2 && b == 1) + } + + static fn knightIndex(df: i32, dr: i32): i32 { + var i = 0 + var found = -1 + for (kdf, kdr) in encKnights() { + if kdf == df && kdr == dr { found = i } + i += 1 + } + return found + } + + static fn dirIndex(sdf: i32, sdr: i32): i32 { + var i = 0 + var found = -1 + for (ddf, ddr) in encDirs() { + if ddf == sdf && ddr == sdr { found = i } + i += 1 + } + return found + } +} From 1c43783bc7a7563e90f1f16ccd98395b41a08cd7 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 23:31:11 +0200 Subject: [PATCH 08/33] =?UTF-8?q?build(polyglot):=20bump=20MSBuild=20trans?= =?UTF-8?q?piler=200.3.1=20=E2=86=92=200.5.3=20+=20fix=20multi-.pg=20incre?= =?UTF-8?q?mental=20bug?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0.5.3 bundles win-x64/linux-x64/linux-arm64/osx-x64/osx-arm64 (adds macOS, so dev there no longer needs a manual $(PolyglotTool)). CI (ubuntu) already worked on 0.3.1 — it too bundled the linux binaries. The version bump does NOT fix the multi-.pg stale-prelude codegen bug (verified: an incremental touch of one .pg still produces CS0101/CS0260 under 0.5.3). Root cause is in the package .targets, not the CLI: PolyglotTranspile uses per-file Inputs/Outputs, so MSBuild's partial-incremental build invokes the CLI with only the changed .pg — but the CLI must see the whole set in one invocation to emit a single shared prelude + `partial class PolyglotProgram`. A single-file run re-emits an inline, non-partial prelude that clashes with the other solvers' generated files. Fix it at our project layer: a _PolyglotForceFullRetranspile target wipes the generated polyglot dir whenever any .pg is newer than the shared prelude, forcing PolyglotTranspile to regenerate the full set in one CLI call. Its single static Output (not a per-file transform) prevents MSBuild from partially batching it. Verified: the incremental touch that reliably broke now rebuilds clean in both Release and Debug; no-op rebuilds stay up-to-date; full suite 362/362, perft 25/25. Durable fix belongs upstream in the .targets (tracked for a MintPlayer.Polyglot PR). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/PLAN.md | 23 +++++++++-------- ....ReinforcementLearning.Environments.csproj | 25 +++++++++++++++---- 2 files changed, 33 insertions(+), 15 deletions(-) diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 1ef6365..ffe5b2f 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1438,16 +1438,19 @@ is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); th - **M40.1 — single-source the engine. ✅ SHIPPED 2026-07-12.** Ported `ChessBoard`/`ChessRules` + `ChessMoveEncoding` into `Environments/Chess/polyglot/chess_solver.pg` (internal `Pg`-prefixed core: `PgChessState` + `PgChessMove` — board `List[64]`, promotion/castling as `i32`, bounded `for`+`continue` rays, `List<(i32,i32)>` delta tables; - every construct proven by `fruitcake_solver.pg`). The `MintPlayer.Polyglot.MSBuild` PackageReference (v0.3.1) already - transpiles every `**/*.pg` to `obj/` before CoreCompile — 0.3.1 already bundles win-x64 + linux-x64 + linux-arm64, - so CI (ubuntu) is unaffected. `ChessState`/`ChessRules`/`ChessMoveEncoding` are now **thin C# facades** over the core - (public API + both test files unchanged); `ChessGame`'s seam (`LegalMoves`/`Apply`/`Result`) delegates to the core's - `legalMoveIndices`/`applyIndex`/`result` so training and the browser share one implementation; `perft` recurses - entirely in-core. **Gate MET: perft 25/25 on the generated engine** (incl. startpos d5 = 4,865,609, Kiwipete - d4 = 4,085,603, ~10 s), encoding round-trip + terminal-detection green, `SelfPlayCampaign` chess contract green, and - the full fast suite 355/355 (no FruitCake/Snake/MountainCar regression from the shared re-transpile). Note: the - multi-`.pg` stale-prelude codegen bug (CS0101/CS0260) reappears on incremental add of a new `.pg` — cleared per the - CLAUDE.md fix (`rm obj/*/net10.0/polyglot/*.cs`) then rebuilt clean. + every construct proven by `fruitcake_solver.pg`). `MintPlayer.Polyglot.MSBuild` transpiles every `**/*.pg` to `obj/` + before CoreCompile (bumped 0.3.1 → **0.5.3**, which bundles win-x64 + linux-x64 + linux-arm64 + osx-x64 + osx-arm64, + so CI/ubuntu is unaffected and macOS dev no longer needs `$(PolyglotTool)`). `ChessState`/`ChessRules`/ + `ChessMoveEncoding` are now **thin C# facades** over the core (public API + both test files unchanged); `ChessGame`'s + seam (`LegalMoves`/`Apply`/`Result`) delegates to the core's `legalMoveIndices`/`applyIndex`/`result` so training and + the browser share one implementation; `perft` recurses entirely in-core. **Gate MET: perft 25/25 on the generated + engine** (incl. startpos d5 = 4,865,609, Kiwipete d4 = 4,085,603, ~10 s), encoding round-trip + terminal-detection + green, `SelfPlayCampaign` chess contract green, full suite 362/362 (no FruitCake/Snake/MountainCar regression from + the shared re-transpile). **Polyglot MSBuild incremental bug (NOT fixed by 0.5.3):** when only one `.pg` changes, + MSBuild's partial-incremental build invokes the CLI with just that file, but the CLI must see the whole set to emit + one shared prelude → a single-file run re-emits an inline non-partial prelude that clashes (CS0101/CS0260). Fixed at + our project layer with a `_PolyglotForceFullRetranspile` target (wipes the generated dir whenever any `.pg` is newer + than the shared prelude, forcing a full-set re-transpile); durable fix belongs upstream in the package `.targets`. - **M40.2 — single-source the inference math + a TS `.ckpt` parser.** Add `PgPolicyValueNet.forward` (flat-array trunk + policy/value heads, `tanh` value), `writeObservation`, and a chess `mcts` (PUCT, masked-softmax, `sqrt`, **no Dirichlet** — inference only) to the `.pg`. Add `ClientApp/src/app/chess/chess-net.ts` (mirrors the C# diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index c81fcff..57a512d 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -4,14 +4,29 @@ - + - + + + + + + From 1da8dc40ec6c51f7909ae20565c62b7e26723816 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 23:37:20 +0200 Subject: [PATCH 09/33] docs(polyglot): sharpen the multi-.pg incremental-bug handoff + verified .targets fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update POLYGLOT_TOPLEVEL_RECORD_BUG.md (the existing Snake-M34 handoff for this bug) with what M40.1 pinned down: - Root cause is the MSBuild .targets, not the CLI — bumping 0.3.1 → 0.5.3 does NOT fix it (verified). PolyglotTranspile's per-file Inputs/Outputs let MSBuild's partial incremental build invoke the CLI with only the CHANGED .pg; the CLI then emits that unit standalone (inline prelude, non-partial PolyglotProgram) and it collides with the other solvers' generated files. The stale prelude is a symptom, not the cause. - Flags that the earlier "clean the --out dir" idea is insufficient and harmful on its own (would delete the other solvers' .cs under a subset invocation). - Gives the recommended upstream fix: a single stamp Output (un-batchable) + RemoveDir, so any staleness re-runs the FULL set in one CLI call. - Documents the consumer-side _PolyglotForceFullRetranspile mitigation already shipped in the Environments csproj, and points the csproj comment at this handoff. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../POLYGLOT_TOPLEVEL_RECORD_BUG.md | 84 ++++++++++++++++++- ....ReinforcementLearning.Environments.csproj | 5 +- 2 files changed, 85 insertions(+), 4 deletions(-) diff --git a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md index 0ac6468..ea0b86d 100644 --- a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md +++ b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md @@ -1,11 +1,16 @@ # Polyglot bug handoff — incremental re-transpile emits a duplicate prelude / non-`partial` PolyglotProgram **Found:** 2026-07-10, during Snake M34 (adding look-ahead search to `snake_solver.pg`). -**Polyglot:** `MintPlayer.Polyglot.MSBuild` **0.3.1** (transpiler `tools/win-x64/polyglot.exe` 0.3.1). -**Polyglot source:** `C:\Repos\MintPlayer.Polyglot` +**Polyglot:** `MintPlayer.Polyglot.MSBuild` **0.3.1** — **still present in 0.5.3** (verified 2026-07-12; see the update at +the bottom). This is an **MSBuild `.targets` bug, not a CLI/version bug**, so bumping the package does not fix it. +**Polyglot source:** `C:\Repos\MintPlayer.Polyglot` (fix lives in `build/MintPlayer.Polyglot.MSBuild.targets`). **Severity:** breaks *incremental* dev builds after editing a `.pg` in a **multi-`.pg` project**; a **clean build always succeeds**, so CI / fresh clones are unaffected — but it's confusing and easy to misdiagnose. +> **⚠️ 2026-07-12 update at the bottom of this file** sharpens the root cause (MSBuild *partial-incremental build* +> passes a **subset** of `.pg` to the CLI) and gives a **verified** `.targets` fix. Note the original "clean the `--out` +> dir" idea in *Fix ideas* below is **insufficient and can be harmful** on its own — read the update first. + ## Symptom In a project with **2+ `.pg` files** built by the glob-all MSBuild target (here: `fruitcake_solver.pg`, @@ -61,3 +66,78 @@ Note: the M34 search *does* use a top-level `record PgSnakeBeamNode` (a typed fr separate TS `any`-inference failure; see `SNAKE_SEARCH_PRD.md` §9). Records build fine on a clean build (FruitCake uses them too); this incremental-staleness bug is unrelated to records. If you hit the errors above during iterative `.pg` editing: `rm src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs` then rebuild. + +--- + +## UPDATE 2026-07-12 — precise root cause + a verified fix (bug persists through 0.5.3) + +Re-encountered during **Chess M40.1** (adding a 4th solver, `chess_solver.pg`). Bumping the package **0.3.1 → 0.5.3** +did **not** fix it — an incremental touch of one `.pg` still produces `CS0101`/`CS0260`/`CS8863`. That rules out the +CLI: **the bug is in the MSBuild `.targets`, and it is version-independent.** + +### The actual trigger (sharper than "stale files in the out dir") + +`PolyglotTranspile` declares a **per-file** output map: + +```xml +Inputs="@(PolyglotFile);$(PolyglotTool)" +Outputs="@(PolyglotFile->'$(PolyglotOutDir)%(Filename).cs')" +``` + +Because both `Inputs` and `Outputs` are item transforms of the same `@(PolyglotFile)`, MSBuild does a **partial +incremental build**: when only *one* `.pg` is newer than its `.cs`, MSBuild runs the target with `@(PolyglotFile)` +**filtered to just that one file**. The `Exec` then invokes the CLI with a **single** source: + +``` +polyglot build "…/chess_solver.pg" --target csharp --out "…/polyglot/." +``` + +From the CLI's point of view that is a **single-module** build, so it (correctly, for a lone module) emits +`chess_solver.cs` in **standalone** mode — inline `Option`/`Some`/`None` + a **non-`partial`** `PolyglotProgram`. That +standalone unit then collides with the **other** solvers' `.cs` + the shared `__polyglot_prelude.cs` still sitting in +`obj/` from the prior full build → `CS0260`/`CS0101`/`CS8863`. The stale prelude is a *symptom*, not the cause; the +cause is **MSBuild handing the CLI a subset**. (`rm *.cs` "fixes" it only because it makes *all* outputs missing, which +forces MSBuild to re-run with the **full** set.) + +### Why the earlier "clean the `--out` dir" idea is not enough — and is harmful alone + +If the CLI cleaned `--out` at the start of every build but MSBuild still invoked it with a **subset**, then a +single-file incremental build would **delete the other solvers' generated `.cs`** and emit only the changed one → +"type `PgFruitCakeWorld` not found" etc. The out-dir cleaning must be paired with **always invoking the CLI with the +full set**. The real fix is therefore MSBuild-side. + +### Verified fix (MSBuild `.targets`) — force all-or-nothing with a single stamp output + +Replace `PolyglotTranspile`'s per-file `Outputs` with a **single stamp file** (a static path, *not* an item transform), +and clean the out dir so a removed/renamed `.pg` leaves no orphan `.cs`. A single static `Outputs` makes the target +**un-batchable**, so any staleness re-runs it with the **complete** `@(PolyglotFile)`: + +```xml + + … (unchanged _PolyglotLibArg / _PolyglotAccessArg) … + + + + + +``` + +`_PolyglotAddGenerated` already globs `$(PolyglotOutDir)*.cs`, so the `.stamp` is not compiled; add it to `FileWrites` +too so `dotnet clean` removes it. A no-op rebuild stays up-to-date (stamp newer than every `.pg` and the tool); any +`.pg`/tool change re-runs the full set. **This is the recommended upstream change.** A defensive +`--clean-out`/idempotent-prelude on the CLI is a fine belt-and-braces addition but is not sufficient on its own (see +above). + +### Consumer-side mitigation already shipped (so this repo no longer needs the `rm`) + +Until the `.targets` ships the fix, `MintPlayer.AI.ReinforcementLearning.Environments.csproj` carries a +`_PolyglotForceFullRetranspile` target that reproduces the effect without editing the package: it wipes +`$(IntermediateOutputPath)polyglot\` whenever any `.pg` is newer than the shared `__polyglot_prelude.cs`, forcing +`PolyglotTranspile` to regenerate the whole set in one CLI call. Verified: the incremental touch that reliably broke now +rebuilds clean in Release **and** Debug; no-op rebuilds stay up-to-date; full test suite 362/362, chess perft 25/25. +When the upstream `.targets` fix lands, that local target can be deleted. diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index 57a512d..9a138a3 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -20,8 +20,9 @@ clashes with the other solvers' generated files (CS0101 duplicate Option/Some/None, CS0260 missing partial). Force all-or-nothing: whenever ANY .pg is newer than the shared prelude, wipe the generated dir so PolyglotTranspile regenerates the FULL set in one CLI call. Single static Output (not a per-file - transform) ⇒ MSBuild can't partially batch this target. (Durable fix belongs upstream in the .targets; - tracked for a MintPlayer.Polyglot PR.) --> + transform) ⇒ MSBuild can't partially batch this target. Durable fix + verified .targets patch: + docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md (2026-07-12 update) — delete this target once it + ships upstream. --> From 9654c9ef5c7c6dff3012a77459ad44cf1f33752c Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Sun, 12 Jul 2026 23:44:09 +0200 Subject: [PATCH 10/33] docs(polyglot): verified drop-in .targets fix for the multi-.pg incremental bug MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Direct writes to C:\Repos\MintPlayer.Polyglot are hook-blocked, so package the fix as a ready-to-apply handoff instead of a direct PR: - Add docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED — the upstream build/MintPlayer.Polyglot.MSBuild.targets with the minimal fix applied (stamp Outputs + RemoveDir + Touch + FileWrites stamp, plus the rationale comment). A session inside the Polyglot repo copies it over the source file, branches, and PRs. - Strengthen the handoff doc: the patch is VERIFIED, not proposed. Built a 2-.pg repro harness that imports the SOURCE .targets directly (no NuGet-cache mutation): the unpatched target fails the touch-one-.pg rebuild (a.cs standalone → CS0101/CS0260/ CS8863); the patched target makes all four states green (recover, incremental touch, no-op up-to-date skip, clean build), with incrementality preserved via the stamp. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../MintPlayer.Polyglot.MSBuild.targets.FIXED | 87 +++++++++++++++++++ .../POLYGLOT_TOPLEVEL_RECORD_BUG.md | 29 ++++++- 2 files changed, 113 insertions(+), 3 deletions(-) create mode 100644 docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED diff --git a/docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED b/docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED new file mode 100644 index 0000000..58485a7 --- /dev/null +++ b/docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED @@ -0,0 +1,87 @@ + + + + + + $(IntermediateOutputPath)polyglot\ + + $(NETCoreSdkPortableRuntimeIdentifier) + <_PolyglotToolName Condition="$([MSBuild]::IsOSPlatform('Windows'))">polyglot.exe + <_PolyglotToolName Condition="'$(_PolyglotToolName)' == ''">polyglot + $(MSBuildThisFileDirectory)..\tools\$(PolyglotHostRid)\$(_PolyglotToolName) + + + + + + + + + + + + + + + + + + <_PolyglotLibArg> + <_PolyglotLibArg Condition="'$(PolyglotLib)' != ''"> --lib "$(PolyglotLib)" + + <_PolyglotAccessArg> + <_PolyglotAccessArg Condition="'$(PolyglotAccess)' != ''"> --access "$(PolyglotAccess)" + + + + + + + + + + + <_PolyglotGenerated Include="$(PolyglotOutDir)*.cs" /> + + + + + + + diff --git a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md index ea0b86d..f2c54db 100644 --- a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md +++ b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md @@ -129,9 +129,32 @@ and clean the out dir so a removed/renamed `.pg` leaves no orphan `.cs`. A singl `_PolyglotAddGenerated` already globs `$(PolyglotOutDir)*.cs`, so the `.stamp` is not compiled; add it to `FileWrites` too so `dotnet clean` removes it. A no-op rebuild stays up-to-date (stamp newer than every `.pg` and the tool); any -`.pg`/tool change re-runs the full set. **This is the recommended upstream change.** A defensive -`--clean-out`/idempotent-prelude on the CLI is a fine belt-and-braces addition but is not sufficient on its own (see -above). +`.pg`/tool change re-runs the full set. A defensive `--clean-out`/idempotent-prelude on the CLI is a fine +belt-and-braces addition but is not sufficient on its own (see above). + +**This exact patch is VERIFIED (2026-07-12), not just proposed.** A drop-in copy of the fixed target file is staged +alongside this doc: **`docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED`** — its only diff vs the +current upstream `build/MintPlayer.Polyglot.MSBuild.targets` is the four lines above (stamp `Outputs`, `RemoveDir`, +`Touch`, `FileWrites` stamp) plus the rationale comment. Session B in `C:\Repos\MintPlayer.Polyglot` can copy it over +`src/MintPlayer.Polyglot.MSBuild/build/MintPlayer.Polyglot.MSBuild.targets`, then branch + PR. + +**How it was verified — a 2-`.pg` repro harness importing the source `.targets` directly** (no NuGet cache mutation): + +```bash +D=/tmp/pgfix; mkdir -p "$D"; cd "$D" +printf 'fn aHello(): i32 => 1\n' > a.pg +printf 'fn bHello(): i32 => 2\n' > b.pg +# Test.csproj: net10.0 project that s the .props then the .targets under test, and sets +# to ~/.nuget/packages/mintplayer.polyglot.msbuild/0.5.3/tools/win-x64/polyglot.exe +dotnet build -c Release # clean build: succeeds +touch a.pg && dotnet build -c Release # UNPATCHED .targets → a.cs emitted standalone → CS0101/CS0260/CS8863 +``` + +Against the **unpatched** source `.targets` the `touch a.pg` rebuild fails exactly as reported (`a.cs:7` non-`partial` +PolyglotProgram + duplicate prelude). Against the **patched** `.targets` (the `.FIXED` file): the same `touch` +rebuild **succeeds**, a no-op rebuild logs *"Skipping target PolyglotTranspile because all output files are +up-to-date"* (incrementality preserved), a touch logs *"Touching …/__polyglot.stamp"* (full re-transpile), and a fully +clean build succeeds. All four states green. ### Consumer-side mitigation already shipped (so this repo no longer needs the `rm`) From b631020a66034bc6eceeff03c0b6eecccc85d626 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Mon, 13 Jul 2026 04:39:21 +0200 Subject: [PATCH 11/33] =?UTF-8?q?build(polyglot):=200.5.3=20=E2=86=92=200.?= =?UTF-8?q?6.0,=20drop=20local=20incremental-bug=20workaround=20(fixed=20u?= =?UTF-8?q?pstream)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MintPlayer.Polyglot.MSBuild 0.6.0 (PR #26) ships the stamp-Outputs + RemoveDir fix for the multi-.pg incremental-transpile bug that was diagnosed and verified here during M40.1. With it upstream, the temporary local _PolyglotForceFullRetranspile target and the MintPlayer.Polyglot.MSBuild.targets.FIXED drop-in are no longer needed — both removed. Verified on 0.6.0 with NO local workaround: clean build green; the incremental single-.pg touch that used to break (CS0101/CS0260) now rebuilds clean; no-op rebuild up-to-date; full suite 362/362, chess perft 25/25. Docs reconciled: CLAUDE.md's "known build failure" note marked fixed in 0.6.0; PLAN M40.1 and the polyglot-pilot handoff marked resolved (kept for root-cause history). Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 20 ++--- docs/prd/PLAN.md | 12 +-- .../MintPlayer.Polyglot.MSBuild.targets.FIXED | 87 ------------------- .../POLYGLOT_TOPLEVEL_RECORD_BUG.md | 5 ++ ....ReinforcementLearning.Environments.csproj | 24 ++--- 5 files changed, 24 insertions(+), 124 deletions(-) delete mode 100644 docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED diff --git a/CLAUDE.md b/CLAUDE.md index 59ce5bb..88c85ae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,18 +27,14 @@ proxies the Angular dev server (`UseAngularCliServer` via `UseSpaImproved` in `P If `dotnet run --project src/RLDemo.Web` fails, read the error before touching anything Angular-related. -**Known: stale Polyglot codegen (`CS0260` "missing partial modifier on PolyglotProgram" / `CS0101` -duplicate `Option`/`Some`/`None` prelude).** This is the documented multi-`.pg` incremental-rebuild -transpiler bug (see `docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`) — the CLI doesn't clean its -`--out` dir, so a stale duplicate lingers in `obj/`. It is unrelated to the frontend and to your changes. -Fix by clearing the stale generated files, then run again: - -```bash -rm -f src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs -dotnet run --project src/RLDemo.Web -``` - -Clean/CI builds are unaffected. Never loop build retries against the stale output. +**Fixed (2026-07-13, Polyglot 0.6.0): the multi-`.pg` incremental-rebuild codegen bug** (`CS0260` "missing +partial modifier on PolyglotProgram" / `CS0101` duplicate `Option`/`Some`/`None` prelude). It was an MSBuild +`.targets` bug — a single-`.pg` edit made MSBuild's partial-incremental build hand the transpiler a subset, +which then emitted a standalone/duplicate prelude. `MintPlayer.Polyglot.MSBuild` **0.6.0** (PR #26, stamp +`Outputs` + `RemoveDir`) re-transpiles the full `.pg` set on any edit, so this no longer occurs. If you somehow +hit it on an **older** package, the manual recovery was +`rm -f src/MintPlayer.AI.ReinforcementLearning.Environments/obj/*/net10.0/polyglot/*.cs` then rebuild. History + +root cause: `docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`. Never loop build retries against stale output. ## Stopping a `dotnet run` host diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index ffe5b2f..b6698fd 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1439,18 +1439,18 @@ is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); th into `Environments/Chess/polyglot/chess_solver.pg` (internal `Pg`-prefixed core: `PgChessState` + `PgChessMove` — board `List[64]`, promotion/castling as `i32`, bounded `for`+`continue` rays, `List<(i32,i32)>` delta tables; every construct proven by `fruitcake_solver.pg`). `MintPlayer.Polyglot.MSBuild` transpiles every `**/*.pg` to `obj/` - before CoreCompile (bumped 0.3.1 → **0.5.3**, which bundles win-x64 + linux-x64 + linux-arm64 + osx-x64 + osx-arm64, + before CoreCompile (bumped 0.3.1 → **0.6.0**, which bundles win-x64 + linux-x64 + linux-arm64 + osx-x64 + osx-arm64, so CI/ubuntu is unaffected and macOS dev no longer needs `$(PolyglotTool)`). `ChessState`/`ChessRules`/ `ChessMoveEncoding` are now **thin C# facades** over the core (public API + both test files unchanged); `ChessGame`'s seam (`LegalMoves`/`Apply`/`Result`) delegates to the core's `legalMoveIndices`/`applyIndex`/`result` so training and the browser share one implementation; `perft` recurses entirely in-core. **Gate MET: perft 25/25 on the generated engine** (incl. startpos d5 = 4,865,609, Kiwipete d4 = 4,085,603, ~10 s), encoding round-trip + terminal-detection green, `SelfPlayCampaign` chess contract green, full suite 362/362 (no FruitCake/Snake/MountainCar regression from - the shared re-transpile). **Polyglot MSBuild incremental bug (NOT fixed by 0.5.3):** when only one `.pg` changes, - MSBuild's partial-incremental build invokes the CLI with just that file, but the CLI must see the whole set to emit - one shared prelude → a single-file run re-emits an inline non-partial prelude that clashes (CS0101/CS0260). Fixed at - our project layer with a `_PolyglotForceFullRetranspile` target (wipes the generated dir whenever any `.pg` is newer - than the shared prelude, forcing a full-set re-transpile); durable fix belongs upstream in the package `.targets`. + the shared re-transpile). **Polyglot MSBuild multi-`.pg` incremental bug — FIXED upstream in 0.6.0 (PR #26):** a + single-`.pg` edit used to make MSBuild's partial-incremental build hand the CLI a subset → inline duplicate prelude + clash (CS0101/CS0260). Diagnosed here, verified a stamp-`Outputs` + `RemoveDir` fix against the source `.targets`, + and it shipped in 0.6.0; the temporary local `_PolyglotForceFullRetranspile` workaround has been removed + (`docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`). - **M40.2 — single-source the inference math + a TS `.ckpt` parser.** Add `PgPolicyValueNet.forward` (flat-array trunk + policy/value heads, `tanh` value), `writeObservation`, and a chess `mcts` (PUCT, masked-softmax, `sqrt`, **no Dirichlet** — inference only) to the `.pg`. Add `ClientApp/src/app/chess/chess-net.ts` (mirrors the C# diff --git a/docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED b/docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED deleted file mode 100644 index 58485a7..0000000 --- a/docs/prd/polyglot-pilot/MintPlayer.Polyglot.MSBuild.targets.FIXED +++ /dev/null @@ -1,87 +0,0 @@ - - - - - - $(IntermediateOutputPath)polyglot\ - - $(NETCoreSdkPortableRuntimeIdentifier) - <_PolyglotToolName Condition="$([MSBuild]::IsOSPlatform('Windows'))">polyglot.exe - <_PolyglotToolName Condition="'$(_PolyglotToolName)' == ''">polyglot - $(MSBuildThisFileDirectory)..\tools\$(PolyglotHostRid)\$(_PolyglotToolName) - - - - - - - - - - - - - - - - - - <_PolyglotLibArg> - <_PolyglotLibArg Condition="'$(PolyglotLib)' != ''"> --lib "$(PolyglotLib)" - - <_PolyglotAccessArg> - <_PolyglotAccessArg Condition="'$(PolyglotAccess)' != ''"> --access "$(PolyglotAccess)" - - - - - - - - - - - <_PolyglotGenerated Include="$(PolyglotOutDir)*.cs" /> - - - - - - - diff --git a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md index f2c54db..28df398 100644 --- a/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md +++ b/docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md @@ -1,5 +1,10 @@ # Polyglot bug handoff — incremental re-transpile emits a duplicate prelude / non-`partial` PolyglotProgram +> **✅ RESOLVED 2026-07-13 in `MintPlayer.Polyglot.MSBuild` 0.6.0 (PR #26).** The verified stamp-`Outputs` + +> `RemoveDir` fix (see the 2026-07-12 update below) shipped upstream. This repo is on 0.6.0; the local +> `_PolyglotForceFullRetranspile` workaround and the `MintPlayer.Polyglot.MSBuild.targets.FIXED` drop-in were +> removed. Kept for history / root-cause reference. + **Found:** 2026-07-10, during Snake M34 (adding look-ahead search to `snake_solver.pg`). **Polyglot:** `MintPlayer.Polyglot.MSBuild` **0.3.1** — **still present in 0.5.3** (verified 2026-07-12; see the update at the bottom). This is an **MSBuild `.targets` bug, not a CLI/version bug**, so bumping the package does not fix it. diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index 9a138a3..16f9982 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -6,28 +6,14 @@ + build-only transpiler OUT of this published package's dependencies. The 0.6.0 CLI is bundled for + win-x64/linux-x64/linux-arm64/osx-x64/osx-arm64 (covers Windows/macOS dev + Linux CI/deploy). 0.6.0 also + fixes the multi-.pg incremental-transpile bug upstream (PR #26 — stamp Outputs + RemoveDir, so a single-.pg + edit re-transpiles the FULL set), so the previous local _PolyglotForceFullRetranspile workaround is gone. --> - + - - - - - From 3a79858de4bf3b9a6823212c9ab30c4c0340f4b3 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Mon, 13 Jul 2026 08:58:35 +0200 Subject: [PATCH 12/33] =?UTF-8?q?M40.2=20=E2=80=94=20single-source=20the?= =?UTF-8?q?=20chess=20inference=20math=20(net=20forward=20+=20MCTS)=20+=20?= =?UTF-8?q?TS=20.ckpt=20parser?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the browser inference path to chess_solver.pg (transpiled to C# + committed TS): - writeObservation(): the 18-plane × 64 = 1152 observation, matching ChessGame.WriteObservation exactly (verified by a parity test over startpos / Kiwipete / an en-passant position). - PgPolicyValueNet.forward(): ReLU trunk (flat-array weights) → policy logits + linear value, mirroring Core/Nn/PolicyValueNet on one row (same shape as fruitcake's PgDuelingNet). - PgChessMcts: inference-only AlphaZero PUCT (masked-softmax priors, sqrt exploration, value negated per ply, NO Dirichlet) — the browser/serving search, single-sourced so C# and TS share it. - Math cleanup: use std.math Math.abs/Math.max (type-preserving on i32), keep isign (no std sign). C# side (Environments): - ChessNetParityTests (the M40.2 gate): Save a PolicyValueNet through its real .ckpt path, parse the bytes into the generated PgPolicyValueNet exactly as chess-net.ts will, and assert forward (logits + value) agrees within an f32 tolerance on the start position. Plus writeObservation parity and an MCTS runtime smoke (valid legal-move distribution summing to 1; chooseMove legal). 358/358 fast tests green. Browser side (RLDemo.Web ClientApp): - chess/chess_solver.ts — committed TS twin (regenerated from the .pg; compiles under the app's tsconfig). - chess/chess-net.ts — the ONE non-single-sourced piece: parses the "selfplay-pv" .ckpt (magic RLNC, trunk widths, per-layer W/b in Parameters() order; inputSize=1152/actions=4672 supplied) into the generated PgPolicyValueNet. Mirrors fruitcake-net.ts. Note: the generated TS is loosely typed in spots (emitter drops local List annotations; renders List as `T | null[]`) — compiles under the app's non-strict tsconfig and is runtime-correct, but the emitter gaps will be handed off to MintPlayer.Polyglot. M40.3 (the interactive board) is next. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../Chess/polyglot/chess_solver.pg | 255 +++++- .../ClientApp/src/app/chess/chess-net.ts | 89 ++ .../ClientApp/src/app/chess/chess_solver.ts | 845 ++++++++++++++++++ .../ChessNetParityTests.cs | 130 +++ 4 files changed, 1311 insertions(+), 8 deletions(-) create mode 100644 src/RLDemo.Web/ClientApp/src/app/chess/chess-net.ts create mode 100644 src/RLDemo.Web/ClientApp/src/app/chess/chess_solver.ts create mode 100644 tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessNetParityTests.cs diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg index bd94fae..3d27ae3 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Chess/polyglot/chess_solver.pg @@ -17,6 +17,7 @@ // construct here is one proven by fruitcake_solver.pg. import { List } from "std.collections" +import { Math } from "std.math" // A move: from/to squares (0..63 = rank*8 + file) plus the promotion piece code // (0 = none) for a pawn reaching the last rank. Castling / en passant / double-push @@ -51,12 +52,10 @@ class PgChessState { this.halfmoveClock = halfmoveClock } - // ── small helpers (i32 arithmetic std.math doesn't cover) ───────────────────────────────── + // ── small helpers (std.math covers abs/max/min type-preserving on i32; only sign is missing) ─ static fn fileOf(sq: i32): i32 => sq % 8 static fn rankOf(sq: i32): i32 => sq / 8 - static fn iabs(x: i32): i32 => if x < 0 { -x } else { x } static fn isign(x: i32): i32 => if x > 0 { 1 } else { if x < 0 { -1 } else { 0 } } - static fn imax(a: i32, b: i32): i32 => if a > b { a } else { b } static fn onBoard(f: i32, r: i32): bool => f >= 0 && f < 8 && r >= 0 && r < 8 static fn isWhite(piece: i32): bool => piece > 0 static fn pieceType(piece: i32): i32 => if piece < 0 { -piece } else { piece } @@ -326,12 +325,12 @@ class PgChessState { } // Double push sets the en-passant target square. - if t == 1 && iabs(rankOf(move.to) - rankOf(move.from)) == 2 { + if t == 1 && Math.abs(rankOf(move.to) - rankOf(move.from)) == 2 { newEp = (move.from + move.to) / 2 } // Castling: move the rook too. - if t == 6 && iabs(fileOf(move.to) - fileOf(move.from)) == 2 { + if t == 6 && Math.abs(fileOf(move.to) - fileOf(move.from)) == 2 { let rank = rankOf(move.from) if fileOf(move.to) == 6 { b[rank * 8 + 5] = b[rank * 8 + 7]; b[rank * 8 + 7] = 0 } // king-side else { b[rank * 8 + 3] = b[rank * 8 + 0]; b[rank * 8 + 0] = 0 } // queen-side @@ -403,6 +402,34 @@ class PgChessState { return out } + // The 18-plane × 64 = 1152 observation the net consumes (single-sourced; the C# facade casts f64→float + // for the float32 net, the browser feeds it directly). Planes: [0–5] White P,N,B,R,Q,K, [6–11] Black, + // [12] side-to-move (all-ones iff white to move), [13–16] castling WK/WQ/BK/BQ (each all-ones iff the + // right is held), [17] en-passant target square (one-hot). Must match ChessGame.WriteObservation exactly. + fn writeObservation(): List { + var obs: List = List() + for i in 0..1152 { obs.add(0.0) } + + for sq in 0..64 { + let piece = this.squares[sq] + if piece == 0 { continue } + let t = pieceType(piece) - 1 // 0..5 + let plane = if piece > 0 { t } else { 6 + t } // white vs black block + obs[plane * 64 + sq] = 1.0 + } + if this.whiteToMove { this.fillPlane(obs, 12) } + if (this.castling & CastleWK) != 0 { this.fillPlane(obs, 13) } + if (this.castling & CastleWQ) != 0 { this.fillPlane(obs, 14) } + if (this.castling & CastleBK) != 0 { this.fillPlane(obs, 15) } + if (this.castling & CastleBQ) != 0 { this.fillPlane(obs, 16) } + if this.enPassant >= 0 { obs[17 * 64 + this.enPassant] = 1.0 } + return obs + } + + fn fillPlane(obs: List, plane: i32) { + for sq in 0..64 { obs[plane * 64 + sq] = 1.0 } + } + // Apply an encoded action index (a queen-promotion rides the queen planes and decodes with // promotion 0 → promote a pawn reaching the last rank to a Queen by default). fn applyIndex(index: i32): PgChessState { @@ -429,7 +456,7 @@ class PgChessState { plane = 56 + knightIndex(df, dr) } else { let dir = dirIndex(isign(df), isign(dr)) - let dist = imax(iabs(df), iabs(dr)) + let dist = Math.max(Math.abs(df), Math.abs(dr)) plane = dir * 7 + (dist - 1) } return move.from * PlanesPerSquare + plane @@ -474,8 +501,8 @@ class PgChessState { } static fn isKnightMove(df: i32, dr: i32): bool { - let a = iabs(df) - let b = iabs(dr) + let a = Math.abs(df) + let b = Math.abs(dr) return (a == 1 && b == 2) || (a == 2 && b == 1) } @@ -499,3 +526,215 @@ class PgChessState { return found } } + +// ── Two-headed policy/value net inference (single-source forward pass) ──────────────────────── +// Mirrors Core/Nn/PolicyValueNet: a ReLU trunk → policy logits (one per action, linear) + a scalar +// value (linear). INFERENCE ONLY — training stays on the SDK autograd/GEMM. Weights are float32 in +// the .ckpt; the per-platform parser (chess-net.ts / the C# parity loader) loads them into these f64 +// flat arrays. Trunk weights/biases are concatenated in layer order (each row-major [in, out]) with +// per-layer offsets from `hidden` — a single List avoids nested-generic (List>) params, +// which the transpiler mishandles (same shape as fruitcake's PgDuelingNet). + +// forward() output: raw policy logits (length actions) + the raw scalar value (linear — the leaf +// evaluator applies tanh, matching PolicyValueNet's linear value head + SelfPlayCampaign.Evaluate). +record PgNetOut(logits: List, value: f64) + +class PgPolicyValueNet { + var inputSize: i32 + var actions: i32 + var hidden: List // trunk hidden widths (from the checkpoint) + var trunkWFlat: List // trunk weight matrices concatenated in layer order (each row-major [in, out]) + var trunkBFlat: List // trunk biases concatenated in layer order + var policyW: List // [lastHidden * actions] + var policyB: List // [actions] + var valueW: List // [lastHidden * 1] + var valueB: List // [1] + + init(inputSize: i32, actions: i32, hidden: List, trunkWFlat: List, trunkBFlat: List, policyW: List, policyB: List, valueW: List, valueB: List) { + this.inputSize = inputSize + this.actions = actions + this.hidden = hidden + this.trunkWFlat = trunkWFlat + this.trunkBFlat = trunkBFlat + this.policyW = policyW + this.policyB = policyB + this.valueW = valueW + this.valueB = valueB + } + + // obs (length inputSize) → logits (length actions) + scalar value. Matches PolicyValueNet.Forward on one row. + fn forward(obs: List): PgNetOut { + var x = obs + var prev = this.inputSize + var wOff = 0 + var bOff = 0 + for l in 0..this.hidden.count { + let h = this.hidden[l] + x = relu(linear(x, this.trunkWFlat, wOff, this.trunkBFlat, bOff, prev, h)) + wOff += prev * h + bOff += h + prev = h + } + let logits = linear(x, this.policyW, 0, this.policyB, 0, prev, this.actions) + let value = linear(x, this.valueW, 0, this.valueB, 0, prev, 1) + return PgNetOut(logits, value[0]) + } + + // Dense layer: out[o] = b[bOff+o] + Σ_i x[i]·w[wOff + i*outDim + o] (w row-major [inDim, outDim]). + static fn linear(x: List, w: List, wOff: i32, b: List, bOff: i32, inDim: i32, outDim: i32): List { + var out: List = List() + for o in 0..outDim { + var s = b[bOff + o] + for i in 0..inDim { s += x[i] * w[wOff + i * outDim + o] } + out.add(s) + } + return out + } + + static fn relu(x: List): List { + var out: List = List() + for v in x { out.add(Math.max(0.0, v)) } + return out + } +} + +// ── Inference MCTS (AlphaZero PUCT) ─────────────────────────────────────────────────────────── +// Single-source port of Core/Planning/Mcts, minus the self-play-only Dirichlet root noise (this is +// the browser/serving path). Each simulation walks the tree maximizing Q + U, expands one leaf, and +// backs the net value up the path NEGATING it every ply (zero-sum). `search` returns the root +// visit-count distribution over the 4672 action space; `chooseMove` is its argmax. + +// A leaf evaluation: priors aligned to the node's legal-move list (masked-softmax over legal) + the +// tanh value from the side-to-move's perspective. +record PgLeafEval(priors: List, value: f64) + +class PgMctsNode { + var moves: List // legal action indices (empty if terminal) + var p: List // prior per move (aligned to moves) + var n: List // visit count per move + var w: List // summed value per move (this node's mover perspective) + var children: List + var expanded: bool + var terminal: bool + var terminalValue: f64 // set when terminal: -1 loss (side-to-move mated) / 0 draw, mover-relative + + init(moves: List, terminal: bool, terminalValue: f64) { + this.moves = moves + this.terminal = terminal + this.terminalValue = terminalValue + this.p = List() + this.n = List() + this.w = List() + this.children = List() + this.expanded = false + } +} + +class PgChessMcts { + // Run `sims` PUCT simulations from `root` and return the root visit-count distribution over the + // 4672 action space (0 for illegal/unvisited). Falls back to raw priors if nothing was visited. + static fn search(net: PgPolicyValueNet, root: PgChessState, sims: i32, cpuct: f64): List { + let rootNode = newNode(root) + if !rootNode.terminal { + expandLeaf(rootNode, net, root) + for s in 0..sims { simulate(rootNode, net, root, cpuct) } + } + + var pi: List = List() + for i in 0..4672 { pi.add(0.0) } + var total = 0 + for i in 0..rootNode.moves.count { total += rootNode.n[i] } + if total == 0 { + for i in 0..rootNode.moves.count { pi[rootNode.moves[i]] = rootNode.p[i] } + return pi + } + for i in 0..rootNode.moves.count { pi[rootNode.moves[i]] = rootNode.n[i] * 1.0 / total } + return pi + } + + // Best action index (argmax visit count via the returned distribution). + static fn chooseMove(net: PgPolicyValueNet, root: PgChessState, sims: i32, cpuct: f64): i32 { + let pi = search(net, root, sims, cpuct) + var best = -1 + var bestV = -1.0 + for i in 0..pi.count { if pi[i] > bestV { bestV = pi[i]; best = i } } + return best + } + + // Returns the value of `state` from the perspective of its side to move. + static fn simulate(node: PgMctsNode, net: PgPolicyValueNet, state: PgChessState, cpuct: f64): f64 { + if node.terminal { return node.terminalValue } + if !node.expanded { return expandLeaf(node, net, state) } + + let edge = selectChild(node, cpuct) + let childState = state.applyIndex(node.moves[edge]) + if node.children[edge] == null { node.children[edge] = newNode(childState) } + let child = node.children[edge]! + let value = -simulate(child, net, childState, cpuct) // flip: child mover's value + node.n[edge] = node.n[edge] + 1 + node.w[edge] = node.w[edge] + value + return value + } + + static fn selectChild(node: PgMctsNode, cpuct: f64): i32 { + var sumN = 0 + for i in 0..node.n.count { sumN += node.n[i] } + let sqrtSum = Math.sqrt(sumN * 1.0) + var best = 0 + var bestScore = -1.0e30 + for i in 0..node.moves.count { + let q = if node.n[i] > 0 { node.w[i] / node.n[i] } else { 0.0 } + let u = cpuct * node.p[i] * sqrtSum / (1.0 + node.n[i]) + let score = q + u + if score > bestScore { bestScore = score; best = i } + } + return best + } + + static fn newNode(state: PgChessState): PgMctsNode { + let r = state.result() // 0 ongoing, 1 loss (side-to-move mated), 2 draw + if r != 0 { + let tv = if r == 1 { -1.0 } else { 0.0 } + return PgMctsNode(List(), true, tv) + } + return PgMctsNode(state.legalMoveIndices(), false, 0.0) + } + + // Evaluate the leaf, seed per-move priors (masked-softmax over legal), return the net's value estimate. + static fn expandLeaf(node: PgMctsNode, net: PgPolicyValueNet, state: PgChessState): f64 { + let ev = evaluate(net, state, node.moves) + let k = node.moves.count + node.p = List() + node.n = List() + node.w = List() + node.children = List() + for i in 0..k { + node.p.add(ev.priors[i]) + node.n.add(0) + node.w.add(0.0) + node.children.add(null) + } + node.expanded = true + return ev.value + } + + // Masked-softmax priors over `moves` (aligned) + tanh value. Mirrors SelfPlayCampaign.Evaluate. + static fn evaluate(net: PgPolicyValueNet, state: PgChessState, moves: List): PgLeafEval { + let out = net.forward(state.writeObservation()) + var mx = -1.0e30 + for m in moves { if out.logits[m] > mx { mx = out.logits[m] } } + var pr: List = List() + var sum = 0.0 + for m in moves { + let e = Math.exp(out.logits[m] - mx) + pr.add(e) + sum += e + } + if sum > 0.0 { + for i in 0..pr.count { pr[i] = pr[i] / sum } + } else { + for i in 0..pr.count { pr[i] = 1.0 / pr.count } + } + return PgLeafEval(pr, Math.tanh(out.value)) + } +} diff --git a/src/RLDemo.Web/ClientApp/src/app/chess/chess-net.ts b/src/RLDemo.Web/ClientApp/src/app/chess/chess-net.ts new file mode 100644 index 0000000..326a0b7 --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/chess/chess-net.ts @@ -0,0 +1,89 @@ +// Loads the self-play-trained chess policy/value checkpoint in the browser and builds the single-source +// PgPolicyValueNet (generated from chess_solver.pg). This is the ONE piece not single-sourced through Polyglot: +// parsing the .ckpt binary is I/O, not pure math. It mirrors the C# writer (PolicyValueNet.Save / CheckpointFormat), +// kind "selfplay-pv": +// +// uint32 magic = "RLNC" (0x434E4C52, little-endian on disk 52 4C 4E 43) +// string kind = "selfplay-pv" (BinaryWriter: 7-bit-encoded-int length prefix + UTF-8 bytes) +// int32 version (= 2) +// int32 trunkCount, then trunkCount × int32 (the trunk hidden widths) +// per parameter, in PolicyValueNet.Parameters() order = [each trunk layer, then policyHead, then valueHead]: +// int32 count + count × float32 (weight, row-major [inDim, outDim]; then bias) +// +// inputSize (1152) and actions (4672) are NOT stored — they come from ChessGame and are passed in. +// A float32 read into a JS number is the exact f64 of that float32, matching the C# `(double)f` load, so the +// browser net is the f64 twin of the training net (agrees to within an f32 tolerance — see ChessNetParityTests). + +import { PgPolicyValueNet } from './chess_solver'; + +const MAGIC = 0x434e4c52; // "RLNC" +const KIND = 'selfplay-pv'; + +export const CHESS_INPUT_SIZE = 18 * 64; // 1152 +export const CHESS_ACTIONS = 4672; // 64 × 73 AlphaZero move encoding + +/** Fetch + parse the shipped checkpoint. Returns null if it's missing/unreadable/incompatible (the caller can + * then disable the AI or fall back, exactly as the server did when no net was present). */ +export async function loadChessNet( + url = '/models/chess.az.ckpt', + inputSize = CHESS_INPUT_SIZE, + actions = CHESS_ACTIONS, +): Promise { + try { + const resp = await fetch(url); + if (!resp.ok) return null; + return parsePolicyValueNet(await resp.arrayBuffer(), inputSize, actions); + } catch { + return null; + } +} + +export function parsePolicyValueNet( + buffer: ArrayBuffer, + inputSize = CHESS_INPUT_SIZE, + actions = CHESS_ACTIONS, +): PgPolicyValueNet { + const dv = new DataView(buffer); + let off = 0; + const i32 = () => { const v = dv.getInt32(off, true); off += 4; return v; }; + const u32 = () => { const v = dv.getUint32(off, true); off += 4; return v; }; + const u8 = () => dv.getUint8(off++); + const read7bit = () => { // BinaryWriter's 7-bit-encoded length prefix + let result = 0, shift = 0, b: number; + do { b = u8(); result |= (b & 0x7f) << shift; shift += 7; } while (b & 0x80); + return result; + }; + const readString = () => { + const len = read7bit(); + const bytes = new Uint8Array(buffer, off, len); + off += len; + return new TextDecoder().decode(bytes); + }; + const readFloats = () => { + const n = i32(); + const a = new Array(n); + for (let i = 0; i < n; i++) { a[i] = dv.getFloat32(off, true); off += 4; } + return a; + }; + + if (u32() !== MAGIC) throw new Error('chess-net: not an RLNC checkpoint'); + const kind = readString(); + if (kind !== KIND) throw new Error(`chess-net: expected kind '${KIND}', got '${kind}'`); + const version = i32(); + if (version < 1 || version > 2) throw new Error(`chess-net: unsupported version ${version}`); + + const trunkCount = i32(); + const hidden: number[] = []; + for (let i = 0; i < trunkCount; i++) hidden.push(i32()); + + const trunkWFlat: number[] = []; + const trunkBFlat: number[] = []; + for (let l = 0; l < trunkCount; l++) { + for (const f of readFloats()) trunkWFlat.push(f); + for (const f of readFloats()) trunkBFlat.push(f); + } + const policyW = readFloats(); const policyB = readFloats(); + const valueW = readFloats(); const valueB = readFloats(); + + return new PgPolicyValueNet(inputSize, actions, hidden, trunkWFlat, trunkBFlat, policyW, policyB, valueW, valueB); +} diff --git a/src/RLDemo.Web/ClientApp/src/app/chess/chess_solver.ts b/src/RLDemo.Web/ClientApp/src/app/chess/chess_solver.ts new file mode 100644 index 0000000..87d3725 --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/chess/chess_solver.ts @@ -0,0 +1,845 @@ +export type Option = { tag: "Some"; value: T } | { tag: "None" }; +export class PgChessMove { + from: number; + to: number; + promotion: number; + constructor(from: number, to: number, promotion: number) { + this.from = from; + this.to = to; + this.promotion = promotion; + } + equals(other: PgChessMove): boolean { + return this.from === other.from && this.to === other.to && this.promotion === other.promotion; + } +} +export class PgNetOut { + logits: number[]; + value: number; + constructor(logits: number[], value: number) { + this.logits = logits; + this.value = value; + } + equals(other: PgNetOut): boolean { + return this.logits === other.logits && this.value === other.value; + } +} +export class PgLeafEval { + priors: number[]; + value: number; + constructor(priors: number[], value: number) { + this.priors = priors; + this.value = value; + } + equals(other: PgLeafEval): boolean { + return this.priors === other.priors && this.value === other.value; + } +} +export class PgChessState { + static readonly PromoKnight: number = 2; + static readonly PromoBishop: number = 3; + static readonly PromoRook: number = 4; + static readonly PromoQueen: number = 5; + static readonly CastleWK: number = 1; + static readonly CastleWQ: number = 2; + static readonly CastleBK: number = 4; + static readonly CastleBQ: number = 8; + static readonly Size: number = 4672; + static readonly PlanesPerSquare: number = 73; + squares: number[]; + whiteToMove: boolean; + castling: number; + enPassant: number; + halfmoveClock: number; + constructor(squares: number[], whiteToMove: boolean, castling: number, enPassant: number, halfmoveClock: number) { + this.squares = squares; + this.whiteToMove = whiteToMove; + this.castling = castling; + this.enPassant = enPassant; + this.halfmoveClock = halfmoveClock; + } + static fileOf(sq: number): number { + return (sq % 8 | 0); + } + static rankOf(sq: number): number { + return (sq / 8 | 0); + } + static isign(x: number): number { + return (x > 0 ? 1 : (x < 0 ? (-1 | 0) : 0)); + } + static onBoard(f: number, r: number): boolean { + return f >= 0 && f < 8 && r >= 0 && r < 8; + } + static isWhite(piece: number): boolean { + return piece > 0; + } + static pieceType(piece: number): number { + return (piece < 0 ? (-piece | 0) : piece); + } + static clearBits(mask: number, bits: number): number { + return (mask - (mask & bits) | 0); + } + static knightDeltas(): [number, number][] { + let d = []; + d.push([1, 2]); + d.push([2, 1]); + d.push([2, (-1 | 0)]); + d.push([1, (-2 | 0)]); + d.push([(-1 | 0), (-2 | 0)]); + d.push([(-2 | 0), (-1 | 0)]); + d.push([(-2 | 0), 1]); + d.push([(-1 | 0), 2]); + return d; + } + static kingDeltas(): [number, number][] { + let d = []; + d.push([1, 0]); + d.push([1, 1]); + d.push([0, 1]); + d.push([(-1 | 0), 1]); + d.push([(-1 | 0), 0]); + d.push([(-1 | 0), (-1 | 0)]); + d.push([0, (-1 | 0)]); + d.push([1, (-1 | 0)]); + return d; + } + static bishopDirs(): [number, number][] { + let d = []; + d.push([1, 1]); + d.push([(-1 | 0), 1]); + d.push([1, (-1 | 0)]); + d.push([(-1 | 0), (-1 | 0)]); + return d; + } + static rookDirs(): [number, number][] { + let d = []; + d.push([1, 0]); + d.push([(-1 | 0), 0]); + d.push([0, 1]); + d.push([0, (-1 | 0)]); + return d; + } + static encDirs(): [number, number][] { + let d = []; + d.push([0, 1]); + d.push([1, 1]); + d.push([1, 0]); + d.push([1, (-1 | 0)]); + d.push([0, (-1 | 0)]); + d.push([(-1 | 0), (-1 | 0)]); + d.push([(-1 | 0), 0]); + d.push([(-1 | 0), 1]); + return d; + } + static encKnights(): [number, number][] { + let d = []; + d.push([1, 2]); + d.push([2, 1]); + d.push([2, (-1 | 0)]); + d.push([1, (-2 | 0)]); + d.push([(-1 | 0), (-2 | 0)]); + d.push([(-2 | 0), (-1 | 0)]); + d.push([(-2 | 0), 1]); + d.push([(-1 | 0), 2]); + return d; + } + clone(): PgChessState { + let sq = []; + for (const v of this.squares) { + sq.push(v); + } + return new PgChessState(sq, this.whiteToMove, this.castling, this.enPassant, this.halfmoveClock); + } + isSquareAttacked(square: number, byWhite: boolean): boolean { + const tf = PgChessState.fileOf(square); + const tr = PgChessState.rankOf(square); + const pawnDir = (byWhite ? 1 : (-1 | 0)); + const pawn = (byWhite ? 1 : (-1 | 0)); + for (let k = 0; k < 2; k++) { + const df = (Math.imul(k, 2) - 1 | 0); + const f = (tf + df | 0); + const r = (tr - pawnDir | 0); + if (PgChessState.onBoard(f, r) && this.squares[(Math.imul(r, 8) + f | 0)] === pawn) { + return true; + } + } + const knight = (byWhite ? 2 : (-2 | 0)); + const kn = PgChessState.knightDeltas(); + for (const [df, dr] of kn) { + const f = (tf + df | 0); + const r = (tr + dr | 0); + if (PgChessState.onBoard(f, r) && this.squares[(Math.imul(r, 8) + f | 0)] === knight) { + return true; + } + } + const king = (byWhite ? 6 : (-6 | 0)); + const kg = PgChessState.kingDeltas(); + for (const [df, dr] of kg) { + const f = (tf + df | 0); + const r = (tr + dr | 0); + if (PgChessState.onBoard(f, r) && this.squares[(Math.imul(r, 8) + f | 0)] === king) { + return true; + } + } + if (this.rayHits(tf, tr, PgChessState.bishopDirs(), byWhite, 3)) { + return true; + } + if (this.rayHits(tf, tr, PgChessState.rookDirs(), byWhite, 4)) { + return true; + } + return false; + } + rayHits(tf: number, tr: number, dirs: [number, number][], byWhite: boolean, slider: number): boolean { + for (const [df, dr] of dirs) { + let found = false; + let blocked = false; + for (let step = 1; step < 8; step++) { + if (blocked) { + continue; + } + const f = (tf + Math.imul(df, step) | 0); + const r = (tr + Math.imul(dr, step) | 0); + if (!PgChessState.onBoard(f, r)) { + blocked = true; + continue; + } + const piece = this.squares[(Math.imul(r, 8) + f | 0)]; + if (piece !== 0) { + if (PgChessState.isWhite(piece) === byWhite) { + const t = PgChessState.pieceType(piece); + if (t === slider || t === 5) { + found = true; + } + } + blocked = true; + } + } + if (found) { + return true; + } + } + return false; + } + kingSquare(white: boolean): number { + const king = (white ? 6 : (-6 | 0)); + for (let sq = 0; sq < 64; sq++) { + if (this.squares[sq] === king) { + return sq; + } + } + return (-1 | 0); + } + inCheck(white: boolean): boolean { + return this.isSquareAttacked(this.kingSquare(white), !white); + } + legalMoves(): PgChessMove[] { + let legal = []; + const pseudo = this.pseudoLegal(); + for (const m of pseudo) { + const next = this.makeMove(m); + if (!next.inCheck(this.whiteToMove)) { + legal.push(m); + } + } + return legal; + } + pseudoLegal(): PgChessMove[] { + let moves = []; + const white = this.whiteToMove; + for (let sq = 0; sq < 64; sq++) { + const piece = this.squares[sq]; + if (piece === 0 || PgChessState.isWhite(piece) !== white) { + continue; + } + const f = PgChessState.fileOf(sq); + const r = PgChessState.rankOf(sq); + const t = PgChessState.pieceType(piece); + if (t === 1) { + this.pawnMoves(sq, f, r, white, moves); + } else { + if (t === 2) { + this.stepMoves(sq, f, r, white, PgChessState.knightDeltas(), moves); + } else { + if (t === 6) { + this.stepMoves(sq, f, r, white, PgChessState.kingDeltas(), moves); + this.castleMoves(white, moves); + } else { + if (t === 3) { + this.slideMoves(sq, f, r, white, PgChessState.bishopDirs(), moves); + } else { + if (t === 4) { + this.slideMoves(sq, f, r, white, PgChessState.rookDirs(), moves); + } else { + if (t === 5) { + this.slideMoves(sq, f, r, white, PgChessState.bishopDirs(), moves); + this.slideMoves(sq, f, r, white, PgChessState.rookDirs(), moves); + } + } + } + } + } + } + } + return moves; + } + stepMoves(sq: number, f: number, r: number, white: boolean, deltas: [number, number][], moves: PgChessMove[]): void { + for (const [df, dr] of deltas) { + const nf = (f + df | 0); + const nr = (r + dr | 0); + if (!PgChessState.onBoard(nf, nr)) { + continue; + } + const to = (Math.imul(nr, 8) + nf | 0); + const target = this.squares[to]; + if (target === 0 || PgChessState.isWhite(target) !== white) { + moves.push(new PgChessMove(sq, to, 0)); + } + } + } + slideMoves(sq: number, f: number, r: number, white: boolean, dirs: [number, number][], moves: PgChessMove[]): void { + for (const [df, dr] of dirs) { + let blocked = false; + for (let step = 1; step < 8; step++) { + if (blocked) { + continue; + } + const nf = (f + Math.imul(df, step) | 0); + const nr = (r + Math.imul(dr, step) | 0); + if (!PgChessState.onBoard(nf, nr)) { + blocked = true; + continue; + } + const to = (Math.imul(nr, 8) + nf | 0); + const target = this.squares[to]; + if (target === 0) { + moves.push(new PgChessMove(sq, to, 0)); + } else { + if (PgChessState.isWhite(target) !== white) { + moves.push(new PgChessMove(sq, to, 0)); + } + blocked = true; + } + } + } + } + pawnMoves(sq: number, f: number, r: number, white: boolean, moves: PgChessMove[]): void { + const dir = (white ? 1 : (-1 | 0)); + const startRank = (white ? 1 : 6); + const lastRank = (white ? 7 : 0); + const one = (Math.imul((r + dir | 0), 8) + f | 0); + if (this.squares[one] === 0) { + this.addPawn(sq, one, (r + dir | 0) === lastRank, moves); + if (r === startRank) { + const two = (Math.imul((r + Math.imul(2, dir) | 0), 8) + f | 0); + if (this.squares[two] === 0) { + moves.push(new PgChessMove(sq, two, 0)); + } + } + } + for (let k = 0; k < 2; k++) { + const df = (Math.imul(k, 2) - 1 | 0); + const nf = (f + df | 0); + const nr = (r + dir | 0); + if (!PgChessState.onBoard(nf, nr)) { + continue; + } + const to = (Math.imul(nr, 8) + nf | 0); + const target = this.squares[to]; + if (target !== 0 && PgChessState.isWhite(target) !== white) { + this.addPawn(sq, to, nr === lastRank, moves); + } else { + if (to === this.enPassant) { + moves.push(new PgChessMove(sq, to, 0)); + } + } + } + } + addPawn(from: number, to: number, promotion: boolean, moves: PgChessMove[]): void { + if (promotion) { + moves.push(new PgChessMove(from, to, PgChessState.PromoQueen)); + moves.push(new PgChessMove(from, to, PgChessState.PromoRook)); + moves.push(new PgChessMove(from, to, PgChessState.PromoBishop)); + moves.push(new PgChessMove(from, to, PgChessState.PromoKnight)); + } else { + moves.push(new PgChessMove(from, to, 0)); + } + } + castleMoves(white: boolean, moves: PgChessMove[]): void { + const rank = (white ? 0 : 7); + const kingSq = (Math.imul(rank, 8) + 4 | 0); + const ownKing = (white ? 6 : (-6 | 0)); + if (this.squares[kingSq] !== ownKing) { + return; + } + if (this.isSquareAttacked(kingSq, !white)) { + return; + } + const kSide = (white ? PgChessState.CastleWK : PgChessState.CastleBK); + const qSide = (white ? PgChessState.CastleWQ : PgChessState.CastleBQ); + const kEmpty = this.squares[(Math.imul(rank, 8) + 5 | 0)] === 0 && this.squares[(Math.imul(rank, 8) + 6 | 0)] === 0; + const kSafe = !this.isSquareAttacked((Math.imul(rank, 8) + 5 | 0), !white) && !this.isSquareAttacked((Math.imul(rank, 8) + 6 | 0), !white); + if ((this.castling & kSide) !== 0 && kEmpty && kSafe) { + moves.push(new PgChessMove(kingSq, (Math.imul(rank, 8) + 6 | 0), 0)); + } + const qEmpty = this.squares[(Math.imul(rank, 8) + 1 | 0)] === 0 && this.squares[(Math.imul(rank, 8) + 2 | 0)] === 0 && this.squares[(Math.imul(rank, 8) + 3 | 0)] === 0; + const qSafe = !this.isSquareAttacked((Math.imul(rank, 8) + 3 | 0), !white) && !this.isSquareAttacked((Math.imul(rank, 8) + 2 | 0), !white); + if ((this.castling & qSide) !== 0 && qEmpty && qSafe) { + moves.push(new PgChessMove(kingSq, (Math.imul(rank, 8) + 2 | 0), 0)); + } + } + makeMove(move: PgChessMove): PgChessState { + let b = []; + for (const v of this.squares) { + b.push(v); + } + const white = this.whiteToMove; + const piece = b[move.from]; + const t = PgChessState.pieceType(piece); + let capture = b[move.to] !== 0; + let newEp = (-1 | 0); + let castling = this.castling; + b[move.from] = 0; + if (t === 1 && move.to === this.enPassant && this.enPassant >= 0) { + const capturedSq = (move.to + (white ? (-8 | 0) : 8) | 0); + b[capturedSq] = 0; + capture = true; + } + if (move.promotion !== 0) { + b[move.to] = Math.imul((white ? 1 : (-1 | 0)), move.promotion); + } else { + b[move.to] = piece; + } + if (t === 1 && ((a) => (a < (a - a) ? -a : a))((PgChessState.rankOf(move.to) - PgChessState.rankOf(move.from) | 0)) === 2) { + newEp = (((move.from + move.to | 0)) / 2 | 0); + } + if (t === 6 && ((a) => (a < (a - a) ? -a : a))((PgChessState.fileOf(move.to) - PgChessState.fileOf(move.from) | 0)) === 2) { + const rank = PgChessState.rankOf(move.from); + if (PgChessState.fileOf(move.to) === 6) { + b[(Math.imul(rank, 8) + 5 | 0)] = b[(Math.imul(rank, 8) + 7 | 0)]; + b[(Math.imul(rank, 8) + 7 | 0)] = 0; + } else { + b[(Math.imul(rank, 8) + 3 | 0)] = b[(Math.imul(rank, 8) + 0 | 0)]; + b[(Math.imul(rank, 8) + 0 | 0)] = 0; + } + } + if (t === 6) { + if (white) { + castling = PgChessState.clearBits(castling, (PgChessState.CastleWK + PgChessState.CastleWQ | 0)); + } else { + castling = PgChessState.clearBits(castling, (PgChessState.CastleBK + PgChessState.CastleBQ | 0)); + } + } + castling = PgChessState.clearBits(castling, PgChessState.cornerRight(move.from)); + castling = PgChessState.clearBits(castling, PgChessState.cornerRight(move.to)); + const halfmove = (t === 1 || capture ? 0 : (this.halfmoveClock + 1 | 0)); + return new PgChessState(b, !white, castling, newEp, halfmove); + } + static cornerRight(sq: number): number { + if (sq === 0) { + return PgChessState.CastleWQ; + } + if (sq === 7) { + return PgChessState.CastleWK; + } + if (sq === 56) { + return PgChessState.CastleBQ; + } + if (sq === 63) { + return PgChessState.CastleBK; + } + return 0; + } + isFiftyMove(): boolean { + return this.halfmoveClock >= 100; + } + isInsufficientMaterial(): boolean { + let knights = 0; + let bishops = 0; + for (const p of this.squares) { + const t = PgChessState.pieceType(p); + if (t === 0 || t === 6) { + continue; + } else { + if (t === 2) { + knights += 1; + } else { + if (t === 3) { + bishops += 1; + } else { + return false; + } + } + } + } + return (knights + bishops | 0) <= 1; + } + result(): number { + if (this.legalMoves().length === 0) { + return (this.inCheck(this.whiteToMove) ? 1 : 2); + } + if (this.isFiftyMove() || this.isInsufficientMaterial()) { + return 2; + } + return 0; + } + perft(depth: number): number { + if (depth === 0) { + return 1; + } + const moves = this.legalMoves(); + if (depth === 1) { + return moves.length; + } + let nodes = 0; + for (const m of moves) { + nodes += this.makeMove(m).perft((depth - 1 | 0)); + } + return nodes; + } + legalMoveIndices(): number[] { + let out = []; + const moves = this.legalMoves(); + for (const m of moves) { + out.push(PgChessState.encode(m)); + } + return out; + } + writeObservation(): number[] { + let obs = []; + for (let i = 0; i < 1152; i++) { + obs.push(0.0); + } + for (let sq = 0; sq < 64; sq++) { + const piece = this.squares[sq]; + if (piece === 0) { + continue; + } + const t = (PgChessState.pieceType(piece) - 1 | 0); + const plane = (piece > 0 ? t : (6 + t | 0)); + obs[(Math.imul(plane, 64) + sq | 0)] = 1.0; + } + if (this.whiteToMove) { + this.fillPlane(obs, 12); + } + if ((this.castling & PgChessState.CastleWK) !== 0) { + this.fillPlane(obs, 13); + } + if ((this.castling & PgChessState.CastleWQ) !== 0) { + this.fillPlane(obs, 14); + } + if ((this.castling & PgChessState.CastleBK) !== 0) { + this.fillPlane(obs, 15); + } + if ((this.castling & PgChessState.CastleBQ) !== 0) { + this.fillPlane(obs, 16); + } + if (this.enPassant >= 0) { + obs[(Math.imul(17, 64) + this.enPassant | 0)] = 1.0; + } + return obs; + } + fillPlane(obs: number[], plane: number): void { + for (let sq = 0; sq < 64; sq++) { + obs[(Math.imul(plane, 64) + sq | 0)] = 1.0; + } + } + applyIndex(index: number): PgChessState { + const m = PgChessState.decode(index); + let chosen = m; + if (m.promotion === 0 && PgChessState.pieceType(this.squares[m.from]) === 1) { + const toRank = PgChessState.rankOf(m.to); + if (toRank === 0 || toRank === 7) { + chosen = new PgChessMove(m.from, m.to, PgChessState.PromoQueen); + } + } + return this.makeMove(chosen); + } + static encode(move: PgChessMove): number { + const df = (PgChessState.fileOf(move.to) - PgChessState.fileOf(move.from) | 0); + const dr = (PgChessState.rankOf(move.to) - PgChessState.rankOf(move.from) | 0); + let plane = 0; + if (move.promotion === PgChessState.PromoKnight || move.promotion === PgChessState.PromoBishop || move.promotion === PgChessState.PromoRook) { + const pieceIdx = (move.promotion === PgChessState.PromoKnight ? 0 : (move.promotion === PgChessState.PromoBishop ? 1 : 2)); + plane = ((64 + Math.imul(pieceIdx, 3) | 0) + ((df + 1 | 0)) | 0); + } else { + if (PgChessState.isKnightMove(df, dr)) { + plane = (56 + PgChessState.knightIndex(df, dr) | 0); + } else { + const dir = PgChessState.dirIndex(PgChessState.isign(df), PgChessState.isign(dr)); + const dist = ((a, b) => (a >= b ? a : b))(((a) => (a < (a - a) ? -a : a))(df), ((a) => (a < (a - a) ? -a : a))(dr)); + plane = (Math.imul(dir, 7) + ((dist - 1 | 0)) | 0); + } + } + return (Math.imul(move.from, PgChessState.PlanesPerSquare) + plane | 0); + } + static decode(index: number): PgChessMove { + const from = (index / PgChessState.PlanesPerSquare | 0); + const plane = (index % PgChessState.PlanesPerSquare | 0); + const f = PgChessState.fileOf(from); + const r = PgChessState.rankOf(from); + if (plane >= 64) { + const p = (plane - 64 | 0); + const promo = ((p / 3 | 0) === 0 ? PgChessState.PromoKnight : ((p / 3 | 0) === 1 ? PgChessState.PromoBishop : PgChessState.PromoRook)); + const df = ((p % 3 | 0) - 1 | 0); + const dr = (r === 6 ? 1 : (-1 | 0)); + return new PgChessMove(from, (Math.imul((r + dr | 0), 8) + ((f + df | 0)) | 0), promo); + } + if (plane >= 56) { + const target = (plane - 56 | 0); + let i = 0; + let rdf = 0; + let rdr = 0; + for (const [df, dr] of PgChessState.encKnights()) { + if (i === target) { + rdf = df; + rdr = dr; + } + i += 1; + } + return new PgChessMove(from, (Math.imul((r + rdr | 0), 8) + ((f + rdf | 0)) | 0), 0); + } + const dirTarget = (plane / 7 | 0); + const dist = ((plane % 7 | 0) + 1 | 0); + let j = 0; + let ddf = 0; + let ddr = 0; + for (const [df, dr] of PgChessState.encDirs()) { + if (j === dirTarget) { + ddf = df; + ddr = dr; + } + j += 1; + } + return new PgChessMove(from, (Math.imul((r + Math.imul(ddr, dist) | 0), 8) + ((f + Math.imul(ddf, dist) | 0)) | 0), 0); + } + static isKnightMove(df: number, dr: number): boolean { + const a = ((a) => (a < (a - a) ? -a : a))(df); + const b = ((a) => (a < (a - a) ? -a : a))(dr); + return a === 1 && b === 2 || a === 2 && b === 1; + } + static knightIndex(df: number, dr: number): number { + let i = 0; + let found = (-1 | 0); + for (const [kdf, kdr] of PgChessState.encKnights()) { + if (kdf === df && kdr === dr) { + found = i; + } + i += 1; + } + return found; + } + static dirIndex(sdf: number, sdr: number): number { + let i = 0; + let found = (-1 | 0); + for (const [ddf, ddr] of PgChessState.encDirs()) { + if (ddf === sdf && ddr === sdr) { + found = i; + } + i += 1; + } + return found; + } +} +export class PgPolicyValueNet { + inputSize: number; + actions: number; + hidden: number[]; + trunkWFlat: number[]; + trunkBFlat: number[]; + policyW: number[]; + policyB: number[]; + valueW: number[]; + valueB: number[]; + constructor(inputSize: number, actions: number, hidden: number[], trunkWFlat: number[], trunkBFlat: number[], policyW: number[], policyB: number[], valueW: number[], valueB: number[]) { + this.inputSize = inputSize; + this.actions = actions; + this.hidden = hidden; + this.trunkWFlat = trunkWFlat; + this.trunkBFlat = trunkBFlat; + this.policyW = policyW; + this.policyB = policyB; + this.valueW = valueW; + this.valueB = valueB; + } + forward(obs: number[]): PgNetOut { + let x = obs; + let prev = this.inputSize; + let wOff = 0; + let bOff = 0; + for (let l = 0; l < this.hidden.length; l++) { + const h = this.hidden[l]; + x = PgPolicyValueNet.relu(PgPolicyValueNet.linear(x, this.trunkWFlat, wOff, this.trunkBFlat, bOff, prev, h)); + wOff += Math.imul(prev, h); + bOff += h; + prev = h; + } + const logits = PgPolicyValueNet.linear(x, this.policyW, 0, this.policyB, 0, prev, this.actions); + const value = PgPolicyValueNet.linear(x, this.valueW, 0, this.valueB, 0, prev, 1); + return new PgNetOut(logits, value[0]); + } + static linear(x: number[], w: number[], wOff: number, b: number[], bOff: number, inDim: number, outDim: number): number[] { + let out = []; + for (let o = 0; o < outDim; o++) { + let s = b[(bOff + o | 0)]; + for (let i = 0; i < inDim; i++) { + s += x[i] * w[((wOff + Math.imul(i, outDim) | 0) + o | 0)]; + } + out.push(s); + } + return out; + } + static relu(x: number[]): number[] { + let out = []; + for (const v of x) { + out.push(((a, b) => (a >= b ? a : b))(0.0, v)); + } + return out; + } +} +export class PgMctsNode { + moves: number[]; + p: number[]; + n: number[]; + w: number[]; + children: PgMctsNode | null[]; + expanded: boolean; + terminal: boolean; + terminalValue: number; + constructor(moves: number[], terminal: boolean, terminalValue: number) { + this.moves = moves; + this.terminal = terminal; + this.terminalValue = terminalValue; + this.p = []; + this.n = []; + this.w = []; + this.children = []; + this.expanded = false; + } +} +export class PgChessMcts { + static search(net: PgPolicyValueNet, root: PgChessState, sims: number, cpuct: number): number[] { + const rootNode = PgChessMcts.newNode(root); + if (!rootNode.terminal) { + PgChessMcts.expandLeaf(rootNode, net, root); + for (let s = 0; s < sims; s++) { + PgChessMcts.simulate(rootNode, net, root, cpuct); + } + } + let pi = []; + for (let i = 0; i < 4672; i++) { + pi.push(0.0); + } + let total = 0; + for (let i = 0; i < rootNode.moves.length; i++) { + total += rootNode.n[i]; + } + if (total === 0) { + for (let i = 0; i < rootNode.moves.length; i++) { + pi[rootNode.moves[i]] = rootNode.p[i]; + } + return pi; + } + for (let i = 0; i < rootNode.moves.length; i++) { + pi[rootNode.moves[i]] = rootNode.n[i] * 1.0 / total; + } + return pi; + } + static chooseMove(net: PgPolicyValueNet, root: PgChessState, sims: number, cpuct: number): number { + const pi = PgChessMcts.search(net, root, sims, cpuct); + let best = (-1 | 0); + let bestV = -1.0; + for (let i = 0; i < pi.length; i++) { + if (pi[i] > bestV) { + bestV = pi[i]; + best = i; + } + } + return best; + } + static simulate(node: PgMctsNode, net: PgPolicyValueNet, state: PgChessState, cpuct: number): number { + if (node.terminal) { + return node.terminalValue; + } + if (!node.expanded) { + return PgChessMcts.expandLeaf(node, net, state); + } + const edge = PgChessMcts.selectChild(node, cpuct); + const childState = state.applyIndex(node.moves[edge]); + if (node.children[edge] === null) { + node.children[edge] = PgChessMcts.newNode(childState); + } + const child = node.children[edge]; + const value = -PgChessMcts.simulate(child, net, childState, cpuct); + node.n[edge] = (node.n[edge] + 1 | 0); + node.w[edge] = node.w[edge] + value; + return value; + } + static selectChild(node: PgMctsNode, cpuct: number): number { + let sumN = 0; + for (let i = 0; i < node.n.length; i++) { + sumN += node.n[i]; + } + const sqrtSum = Math.sqrt(sumN * 1.0); + let best = 0; + let bestScore = -1.0e30; + for (let i = 0; i < node.moves.length; i++) { + const q = (node.n[i] > 0 ? node.w[i] / node.n[i] : 0.0); + const u = cpuct * node.p[i] * sqrtSum / (1.0 + node.n[i]); + const score = q + u; + if (score > bestScore) { + bestScore = score; + best = i; + } + } + return best; + } + static newNode(state: PgChessState): PgMctsNode { + const r = state.result(); + if (r !== 0) { + const tv = (r === 1 ? -1.0 : 0.0); + return new PgMctsNode([], true, tv); + } + return new PgMctsNode(state.legalMoveIndices(), false, 0.0); + } + static expandLeaf(node: PgMctsNode, net: PgPolicyValueNet, state: PgChessState): number { + const ev = PgChessMcts.evaluate(net, state, node.moves); + const k = node.moves.length; + node.p = []; + node.n = []; + node.w = []; + node.children = []; + for (let i = 0; i < k; i++) { + node.p.push(ev.priors[i]); + node.n.push(0); + node.w.push(0.0); + node.children.push(null); + } + node.expanded = true; + return ev.value; + } + static evaluate(net: PgPolicyValueNet, state: PgChessState, moves: number[]): PgLeafEval { + const out = net.forward(state.writeObservation()); + let mx = -1.0e30; + for (const m of moves) { + if (out.logits[m] > mx) { + mx = out.logits[m]; + } + } + let pr = []; + let sum = 0.0; + for (const m of moves) { + const e = Math.exp(out.logits[m] - mx); + pr.push(e); + sum += e; + } + if (sum > 0.0) { + for (let i = 0; i < pr.length; i++) { + pr[i] = pr[i] / sum; + } + } else { + for (let i = 0; i < pr.length; i++) { + pr[i] = 1.0 / pr.length; + } + } + return new PgLeafEval(pr, Math.tanh(out.value)); + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessNetParityTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessNetParityTests.cs new file mode 100644 index 0000000..adc7336 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/ChessNetParityTests.cs @@ -0,0 +1,130 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Environments.Chess; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// +/// M40.2 gate: the single-source browser inference net (chess_solver.pg → PgPolicyValueNet, transpiled +/// to C# here and to TypeScript for the web client) must compute the SAME forward pass as the training net +/// (). We serialize a real net through its actual .ckpt path, parse the bytes into +/// the generated PgPolicyValueNet exactly as chess-net.ts does in the browser, and compare logits + +/// value on a fixed position. The C# net accumulates in float32 and the generated net in f64 (loaded from the same +/// float32 weights), so they agree only to within an f32 tolerance — which is all the browser AI needs. +/// The byte-level parser here doubles as the reference for chess-net.ts: same magic/kind/order. +/// +public class ChessNetParityTests +{ + private const int InputSize = 18 * 64; // 1152 + private static readonly int Actions = ChessMoveEncoding.Size; // 4672 + + [Fact] + public void Generated_net_matches_PolicyValueNet_forward_within_f32_tolerance() + { + int[] hidden = [24, 16]; + var net = new PolicyValueNet(InputSize, hidden, Actions, new Xoshiro256StarStar(12345)); + + // Round-trip through the real checkpoint format, then rebuild the generated net from the bytes. + using var ms = new MemoryStream(); + net.Save(ms, "selfplay-pv"); + ms.Position = 0; + var pg = LoadPg(ms, InputSize, Actions); + + // Same start-position observation into both nets. + var obs = new float[InputSize]; + new ChessGame().WriteObservation(ChessState.StartPosition(), obs); + + var (logits, value) = net.Forward(new Tensor(obs, 1, InputSize)); + var pgOut = pg.forward([.. obs.Select(f => (double)f)]); + + Assert.Equal(Actions, pgOut.logits.Count); + double maxLogitDiff = 0; + for (int m = 0; m < Actions; m++) + maxLogitDiff = Math.Max(maxLogitDiff, Math.Abs(logits.Data[m] - pgOut.logits[m])); + + Assert.True(maxLogitDiff < 1e-3, $"max logit diff {maxLogitDiff} exceeds f32 tolerance"); + Assert.True(Math.Abs(value.Data[0] - pgOut.value) < 1e-3, $"value diff {Math.Abs(value.Data[0] - pgOut.value)}"); + } + + [Fact] + public void Generated_mcts_returns_a_valid_legal_move_distribution() + { + // The inference MCTS (PgChessMcts) is new single-source code the forward-parity test doesn't exercise. + // Runtime check via the generated C# twin (mirrors the browser TS): search must return a distribution over + // the 4672 space that sums to 1 and puts mass ONLY on legal moves; chooseMove must pick a legal move. + var net = new PolicyValueNet(InputSize, [16], Actions, new Xoshiro256StarStar(7)); + using var ms = new MemoryStream(); + net.Save(ms, "selfplay-pv"); + ms.Position = 0; + var pg = LoadPg(ms, InputSize, Actions); + + var root = ChessState.StartPosition().Core; + var legal = new HashSet(root.legalMoveIndices()); + Assert.Equal(20, legal.Count); // 20 legal moves from the start position + + var pi = PgChessMcts.search(pg, root, 24, 1.25); + Assert.Equal(Actions, pi.Count); + double sum = 0; + for (int i = 0; i < pi.Count; i++) + { + sum += pi[i]; + if (pi[i] > 0) Assert.Contains(i, legal); // no mass on illegal moves + } + Assert.True(Math.Abs(sum - 1.0) < 1e-9, $"visit distribution sums to {sum}, expected 1"); + Assert.Contains(PgChessMcts.chooseMove(pg, root, 24, 1.25), legal); + } + + [Fact] + public void Generated_writeObservation_matches_ChessGame_WriteObservation() + { + // The single-source observation (used by the browser net) must be identical to the training one. + foreach (var fen in new[] + { + ChessFen.StartFen, + "r3k2r/p1ppqpb1/bn2pnp1/3PN3/1p2P3/2N2Q1p/PPPBBPPP/R3K2R w KQkq - 0 1", // Kiwipete (castling rights) + "rnbqkbnr/ppp1p1pp/8/3pPp2/8/8/PPPP1PPP/RNBQKBNR w KQkq f6 0 3", // en-passant target set + }) + { + var state = ChessFen.Parse(fen); + var expected = new float[InputSize]; + new ChessGame().WriteObservation(state, expected); + var actual = state.Core.writeObservation(); + + Assert.Equal(InputSize, actual.Count); + for (int i = 0; i < InputSize; i++) + Assert.Equal(expected[i], (float)actual[i]); + } + } + + // Mirror of chess-net.ts / PolicyValueNet.Save (kind "selfplay-pv"): magic → kind → version → trunk widths → + // per layer [trunk…, policyHead, valueHead] (wCount+floats, bCount+floats). inputSize/actions supplied externally. + private static PgPolicyValueNet LoadPg(Stream stream, int inputSize, int actions) + { + using var r = new BinaryReader(stream, Encoding.UTF8, leaveOpen: true); + Assert.Equal(0x434E4C52u, r.ReadUInt32()); // "RLNC" + Assert.Equal("selfplay-pv", r.ReadString()); + Assert.Equal(2, r.ReadInt32()); // version + + int trunkCount = r.ReadInt32(); + var hidden = new List(trunkCount); + for (int i = 0; i < trunkCount; i++) hidden.Add(r.ReadInt32()); + + List ReadFloats() + { + int n = r.ReadInt32(); + var list = new List(n); + for (int i = 0; i < n; i++) list.Add(r.ReadSingle()); + return list; + } + + var trunkW = new List(); + var trunkB = new List(); + for (int l = 0; l < trunkCount; l++) { trunkW.AddRange(ReadFloats()); trunkB.AddRange(ReadFloats()); } + var policyW = ReadFloats(); var policyB = ReadFloats(); + var valueW = ReadFloats(); var valueB = ReadFloats(); + + return new PgPolicyValueNet(inputSize, actions, hidden, trunkW, trunkB, policyW, policyB, valueW, valueB); + } +} From 9cc771f109649bfff384351911a16fbb5a547fa9 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Mon, 13 Jul 2026 09:06:34 +0200 Subject: [PATCH 13/33] docs(M40.2): mark shipped; note Polyglot TS-emitter issue #27 (verified fix filed) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/PLAN.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index b6698fd..19df051 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1451,11 +1451,18 @@ is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); th clash (CS0101/CS0260). Diagnosed here, verified a stamp-`Outputs` + `RemoveDir` fix against the source `.targets`, and it shipped in 0.6.0; the temporary local `_PolyglotForceFullRetranspile` workaround has been removed (`docs/prd/polyglot-pilot/POLYGLOT_TOPLEVEL_RECORD_BUG.md`). -- **M40.2 — single-source the inference math + a TS `.ckpt` parser.** Add `PgPolicyValueNet.forward` (flat-array - trunk + policy/value heads, `tanh` value), `writeObservation`, and a chess `mcts` (PUCT, masked-softmax, `sqrt`, - **no Dirichlet** — inference only) to the `.pg`. Add `ClientApp/src/app/chess/chess-net.ts` (mirrors the C# - `PolicyValueNet` reader — see the PRD appendix for the exact bytes). **Gate:** C#-vs-generated parity within f32 - tol on a fixed position; committed `chess_solver.ts` emitted. +- **M40.2 — single-source the inference math + a TS `.ckpt` parser. ✅ SHIPPED 2026-07-13.** Added to the `.pg`: + `writeObservation()` (18-plane × 64 = 1152, parity-tested vs `ChessGame.WriteObservation`), `PgPolicyValueNet.forward` + (flat-array ReLU trunk → policy logits + linear value), and `PgChessMcts` (inference PUCT — masked-softmax priors, + `sqrt`, value negated per ply, **no Dirichlet**). `ClientApp/src/app/chess/chess-net.ts` parses the `selfplay-pv` + `.ckpt` (magic RLNC, trunk widths, per-layer W/b in `Parameters()` order; inputSize/actions supplied) into the + generated `PgPolicyValueNet`; committed `chess_solver.ts` emitted. **Gate MET:** `ChessNetParityTests` — C# + `PolicyValueNet.Forward` vs the generated net agree within f32 tol on the start position (round-tripped through the + real `.ckpt` bytes), plus observation parity + an MCTS runtime smoke (valid legal-move distribution, `chooseMove` + legal); 358/358 fast tests. Also did the `std.math` cleanup (`Math.abs`/`Math.max`, kept `isign`). **Known:** the + generated TS is loosely typed in spots (emitter drops local `List` annotations → `let d = []`; renders `List` as + `T | null[]`) — compiles under the app's non-strict `tsconfig` and is runtime-correct; filed with a verified two-part + fix as **MintPlayer.Polyglot issue #27**. - **M40.3 — the browser page.** `chess-director.ts` (runs the transpiled `mcts`+net over the loaded `.ckpt`, like `fruit-cake-director.ts`) + an Angular chess component (board, click-to-move validated by the transpiled `legalMoves`, AI reply, check/mate/draw, last-move highlight) + route/nav; ship `wwwroot/models/chess.az.ckpt` From 061647e46a025398b718dd53b99f1fb725f34e18 Mon Sep 17 00:00:00 2001 From: PieterjanDeClippel Date: Mon, 13 Jul 2026 11:12:53 +0200 Subject: [PATCH 14/33] =?UTF-8?q?build(polyglot):=200.6.0=20=E2=86=92=200.?= =?UTF-8?q?7.0=20=E2=80=94=20TS=20emitter=20fix=20(issue=20#27/PR=20#28);?= =?UTF-8?q?=20chess=5Fsolver.ts=20now=20strictly=20typed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0.7.0 ships the TypeScript-emitter fix diagnosed + verified during M40.2 (typed local List declarations; parenthesized List). Regenerated the committed chess_solver.ts: - `let d: [number, number][] = []` (was `let d = []`) - `let moves: PgChessMove[] = []` - `children: (PgMctsNode | null)[]` (was `PgMctsNode | null[]`) Verified strict-tsc-clean (strict + noImplicitAny) on chess_solver.ts + chess-net.ts — the M40.2 "loosely typed generated TS" caveat is resolved. Full build green, 365/365 tests (incl. deep chess perft) — no regression across the four solvers from the re-transpile. PLAN M40.2 note updated. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/prd/PLAN.md | 9 ++--- ....ReinforcementLearning.Environments.csproj | 10 +++--- .../ClientApp/src/app/chess/chess_solver.ts | 34 +++++++++---------- 3 files changed, 27 insertions(+), 26 deletions(-) diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 19df051..c62ccf5 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1459,10 +1459,11 @@ is a server `ChessController` (per-viewer CPU — the thing M32/M33 removed); th generated `PgPolicyValueNet`; committed `chess_solver.ts` emitted. **Gate MET:** `ChessNetParityTests` — C# `PolicyValueNet.Forward` vs the generated net agree within f32 tol on the start position (round-tripped through the real `.ckpt` bytes), plus observation parity + an MCTS runtime smoke (valid legal-move distribution, `chooseMove` - legal); 358/358 fast tests. Also did the `std.math` cleanup (`Math.abs`/`Math.max`, kept `isign`). **Known:** the - generated TS is loosely typed in spots (emitter drops local `List` annotations → `let d = []`; renders `List` as - `T | null[]`) — compiles under the app's non-strict `tsconfig` and is runtime-correct; filed with a verified two-part - fix as **MintPlayer.Polyglot issue #27**. + legal); 358/358 fast tests. Also did the `std.math` cleanup (`Math.abs`/`Math.max`, kept `isign`). The two TS-emitter + gaps chess first hit (local `List` decls losing their annotation; `List` → `T | null[]`) were filed as + **MintPlayer.Polyglot issue #27** with a verified two-part fix, **fixed upstream in 0.7.0 (PR #28)**; the Environments + `.csproj` is on 0.7.0 and the regenerated `chess_solver.ts` is now **strictly typed** (`let d: [number,number][] = []`, + `children: (PgMctsNode | null)[]`) — verified strict-`tsc`-clean. 365/365 tests on 0.7.0. - **M40.3 — the browser page.** `chess-director.ts` (runs the transpiled `mcts`+net over the loaded `.ckpt`, like `fruit-cake-director.ts`) + an Angular chess component (board, click-to-move validated by the transpiled `legalMoves`, AI reply, check/mate/draw, last-move highlight) + route/nav; ship `wwwroot/models/chess.az.ckpt` diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj index 16f9982..435a3d1 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/MintPlayer.AI.ReinforcementLearning.Environments.csproj @@ -6,12 +6,12 @@ + build-only transpiler OUT of this published package's dependencies. The 0.7.0 CLI is bundled for + win-x64/linux-x64/linux-arm64/osx-x64/osx-arm64 (covers Windows/macOS dev + Linux CI/deploy). 0.6.0 fixed the + multi-.pg incremental-transpile bug (PR #26); 0.7.0 fixes the TypeScript emitter (PR #28 / issue #27 — typed + local List declarations + parenthesized List), so the regenerated chess_solver.ts is strictly typed. --> - +