M23: Pendulum + SAC — continuous control (first Box action space)

docs/ADDING_A_GAME.md

@@ -2,11 +2,18 @@
 
 The mechanical, file-by-file checklist for adding a game to the playground, reverse-engineered from
 **Rush Hour** / **2048** / **Cube** (investigated 2026-06-15). Cross-refs: `PRD.md` §7 + §7.1 (interaction
-models), `PLAN.md` M8–M10 (web slices) + M22 (MountainCar/Snake).
+models), `PLAN.md` M8–M10 (web slices) + M22 (MountainCar/Snake) + M23 (Pendulum/SAC).
 
 For a new game `X` (lowercase env id `xgame`, PascalCase `XGame`), pick the **interaction principle** first
 (PRD §7.1): **A — compute-and-return** (HTTP, like Cube/2048/RushHour/Snake) or **B — live control stream**
-(WebSocket, like MountainCar). Then work the six layers.
+(WebSocket, like MountainCar/Pendulum). Then work the six layers.
+
+**Continuous-action games** (a real-valued action, like Pendulum) differ only at a few seams: the env is
+`IEnvironment<float[],float[]>` with a **`BoxSpace` action space** (not `DiscreteSpace`), the agent is a
+`ContinuousPolicyAgent` over a Gaussian policy net (greedy = tanh(mean), rescaled to the box bounds), trained with
+**SAC** (`SacTrainer`), and the WS `EpisodeStreamer` is used with `TAct = float[]` (pass a zero-vector
+`resetAction`). Serve-time checkpoint is the actor `Mlp` via `MlpCheckpoint` (critics/temperature are training-only).
+Human play maps ←/→ to a continuous torque rather than a discrete button.
 
 ## Load-bearing conventions
 - **Model-store filename:** `FileModelStore.PathOf` maps `(environmentId, algorithmId)` → `<root>/<envId>.<algoId>.ckpt`.

docs/PLAN.md

@@ -781,6 +781,40 @@ page with two modes (watch-AI WS client + human-play TS engine on a JS timer) +
 **Effort:** the first WS streamer is the new infra (~1 d, reused by both); then Snake ~1–2 d, MountainCar ~2–3 d
 (env + PPO). Land the reusable streamer + Snake first (simpler env), then MountainCar.
 
+## M23 — Pendulum + SAC  *(SDK breadth: first continuous control; designed 2026-06-15 by a 4-agent investigation, built same day)*
+
+The SDK's first **continuous-action** algorithm — every prior agent picks one of N discrete actions; SAC outputs a
+real-valued torque. ✅ **BUILT + MEASURED 2026-06-15.** The four-agent investigation found the core is far more
+ready than it looks: `IEnvironment<TObs,TAct>`, `IAgent`, `Evaluator` are already generic over the action type,
+`BoxSpace` is already a valid action space, and the autograd layer needs **zero new backend kernels**.
+
+**New Core machinery (the continuous pillar):**
+- `Tensor.ConcatCols` / `SliceCols` — two small column-wise autograd ops (pure memory layout): `Concat` feeds the
+  critic its `concat(state, action)` input (routing the actor's gradient through the action half); `Slice` splits a
+  policy net's `[B, 2·A]` output into the mean and log-σ halves.
+- `Nn/Normal.cs` — diagonal **squashed Gaussian** (the continuous mirror of `Categorical`): reparameterized
+  `tanh(mean + σ·ε)` sample, log-prob with the tanh change-of-variables correction, `Mode` = tanh(mean). `ε` is a
+  detached `Tensor.RandomNormal` leaf (stop-gradient for free); log-σ clamped to [−20, 2] before exp.
+- `Training/ContinuousReplayBuffer.cs` (+ checkpoint) — sibling of `ReplayBuffer` with float-vector actions and no
+  mask, leaving the discrete buffer byte-compatible.
+- `Training/SacTrainer.cs` — **SAC** (Haarnoja 2018): a squashed-Gaussian actor `[obs → 2·A]`, **twin Q-critics**
+  (plain `Mlp([obs+A,…,1])`, clipped double-Q targets), **Polyak soft target update** (a `SoftUpdate` helper, NOT
+  an overload of the hard `CopyFrom`), and an **auto-tuned entropy temperature** (log-α against target entropy −A).
+  `ContinuousPolicyAgent : IAgent<float[],float[]>` serves greedy = tanh(mean), rescaled to the box bounds. The
+  off-policy collection loop mirrors `DqnTrainer` (single env + replay, warmup, eval cadence, solve-threshold,
+  bitwise-resumable `SacTrainingState`).
+
+**PendulumEnv** (`Environments/PendulumEnv.cs`) — faithful Gymnasium Pendulum-v1: obs `[cos θ, sin θ, θ̇]` (Box 3),
+action torque ∈ [−2,2] (**Box 1**), reward `−(θ² + 0.1·θ̇² + 0.001·τ²)` from the pre-update angle, **semi-implicit
+Euler** (θ advances using the new θ̇), torque clamped (not thrown), **no terminal — truncate@200**. `IStatefulEnvironment`.
+
+**Measured:** SAC (`Hidden [128,128], 30k steps, ~6 min CPU`) reaches greedy eval **mean return −149.1 over 100
+episodes** (random ≈ −1200; past the −200 "solved" bar and the −150 solve-threshold). Bitwise-resume test green.
+Seed shipped to `models/pendulum.sac.ckpt` (Git LFS). **Web (PRD §7.1):** `WS /api/pendulum/live` streams
+`{cosTheta, sinTheta, angularVelocity, torque, reward, done}` (the `EpisodeStreamer` was generalized over `TAct`);
+Angular `<canvas>` rod + **watch-AI** (WS) / **swing-it-yourself** (client TS physics on a JS timer, ←/→ = continuous
+torque). Serve-time checkpoint = actor-only via the existing `MlpCheckpoint` (critics/temperature are training-only).
+
 ## Testing strategy (cross-cutting, from research)
 
 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket.
@@ -824,6 +858,8 @@ page with two modes (watch-AI WS client + human-play TS engine on a JS timer) +
 | Learning-curve levers P.7/P.8/P.9 (2026-06-14) | feed GPU, decouple pacing, per-update efficiency | parallel successor gen (GPU 0→95-100% util) + `--batch`; sample-paced curriculum + lighter eval; ε-loss target sync (freeze-proof) + `--lr`. Net: **~3,050 samples/s (~2× prior)**, GPU-bound at ~2 TFLOP/s |
 | General `IComputeBackend` port Phase 1 (2026-06-14) | every autograd op behind the seam; CPU bitwise-identical | all ops (Map/Zip/reductions/LogSoftmax/Gather/Huber/LayerNorm) routed through the backend; 224/224 green. Phase 2 (device-backed Tensor) parked far-future (no measured win at our scale) |
 | **M21 shortest-move solver — BWAS capability (2026-06-14, residual 1024×4, ~44k iters)** | **provably-optimal shallow; beat Kociemba's QTM mid-range** | batched A* (w=2.5, ≤40k exp, 12/depth): **12/12 & QTM-optimal through depth 10** (exactly d·qt), **100% solved through depth 12**, 83% d13, 75% d14; **every solve beats Kociemba's QTM, often ~2×** (d10 10 vs 19, d12 12.2 vs 29.3). Greedy (live curve) collapses ~d10-11 — search reads the net far deeper. Beats the earlier 1024×3 MLP at every deep level (d12 100% vs 80%). Full god's-number (26 QTM) NOT reached — honest two-tier story |
+| M22 MountainCar (PPO) + Snake (masked Double+Dueling DQN) (2026-06-15) | MountainCar mean ≥ −110; Snake ≥ 5 food | MountainCar **−107.9, goal 100/100** (~120k steps, <1 min); Snake **22.1 food on 12×12** (trained on 6×6 → transfer) |
+| M23 SAC continuous control (2026-06-15) | Pendulum mean ≥ −200 (gate median/3 ≥ −250) | **−149.1 over 100 eps** (Hidden [128,128], 30k steps, ~6 min CPU); random ≈ −1200; bitwise-resume green |
 
 ## Shipped (2026-06-11) — release engineering
 
@@ -896,8 +932,9 @@ curriculum + lighter eval) and **✅ P.9** (LR scaling + ε-loss target sync). F
 3. **AlphaZero-style fine-tune** — *started 2026-06-11, paused mid-campaign; see the
    "Fine-tune round" section under M11 for results so far and the resume command.*
 3. **SDK breadth** (the demos exist to show range): algorithm coverage (✅ PPO action
-   masking *(done)*, ✅ dueling head *(done)*, maybe SAC), more envs (**MountainCar + Snake — designed, see M22**),
-   a TensorBoard writer, public-API stability/semver discipline, reproducibility guarantees.
+   masking *(done)*, ✅ dueling head *(done)*, ✅ SAC continuous control *(done, see M23)*), more envs
+   (✅ **MountainCar + Snake — M22**, ✅ **Pendulum continuous — M23**), a TensorBoard writer, public-API
+   stability/semver discipline, reproducibility guarantees.
 
    **"Switch algorithm, keep the work" — make the three reusable assets first-class.**
    When an algorithm plateaus, what carries over is the env (already portable via
@@ -925,7 +962,7 @@ curriculum + lighter eval) and **✅ P.9** (LR scaling + ε-loss target sync). F
 
 Run the playground: `dotnet run --project src/RLDemo.Web` (Development spawns + proxies
 the Angular dev server itself — do not run `ng serve`). Console demos:
-`dotnet run --project src/RLDemo.Console -c Release -- [grid|lake|cartpole|ppo|2048|2048dqn|rushhour|cube]
+`dotnet run --project src/RLDemo.Console -c Release -- [grid|lake|cartpole|ppo|2048|2048dqn|rushhour|cube|pendulum]
 [seed] [--load] [--save] [--data <dir>]`. Tests: `dotnet test` (`Category=Slow` for gates).
 Training campaigns (resume net + Adam + full training state from the model store):
 `dotnet run --project tools/MintPlayer.AI.ReinforcementLearning.Lab -c Release --

docs/PRD.md

@@ -34,7 +34,8 @@ out (vectorized environments) without rewrites.
 2. **Gymnasium-faithful environment API** for .NET — typed `Reset`/`Step` with separate
    `Terminated`/`Truncated`, spaces, seeding — so results are comparable to literature.
 3. **Algorithm ladder**: tabular Q-learning/SARSA → REINFORCE → DQN (+Double/Dueling) → PPO
-   (vectorized envs, GAE). Each gated by a known-solved threshold before the next starts.
+   (vectorized envs, GAE) → SAC (off-policy continuous control). Each gated by a known-solved
+   threshold before the next starts.
 4. **From-scratch numerics**: own tensor type, SIMD matmul, tape-based autograd, Adam —
    zero native dependencies in the core.
 5. **Real games as benchmarks**: CartPole (literature-comparable flagship), **2048** and
@@ -69,7 +70,9 @@ Explicitly out of scope to prevent the scope creep every research thread warned
 - Distributed training, ONNX export, model-based / offline RL
 - Unity/Godot adapters, netstandard multi-targeting
   *(NuGet publishing was originally out of scope here but shipped on 2026-06-11)*
-- LunarLander or anything needing a physics engine (Box2D port ≈ a project in itself)
+- LunarLander or anything needing a physics engine (Box2D port ≈ a project in itself).
+  *Classic-control continuous tasks (Pendulum) ARE in scope — their dynamics are a trivial ODE;
+  it's only the Box2D-class physics engines that stay out.*
 
 ## 4. Key decisions
 
@@ -136,6 +139,7 @@ Pre-registered, objective "solved" definitions (fixed now so benchmarks stay mea
 | **2048** (4×4, spawn 90% 2 / 10% 4, invalid moves masked) | Box(16) log2-encoded / Discrete(4) | Pre-registered: reach 2048 tile in ≥ 10% of 100 eval games (DQN target); stretch: TD/n-tuple agent ≥ 80% | Owner's game #1; stochastic, showy |
 | **Rush Hour** (6×6 sliding-block; action = (vehicle, ±1 slide); reward −1/step, +100 exit; per-difficulty puzzle sets) | Box(72): vehicle-identity plane + red-car plane / Discrete(32 masked) | ≥ 90% of *easy* puzzle set solved within 2× optimal moves (optimal via built-in BFS solver) | Owner's game #2; sparse-reward planning, curriculum learning |
 | MountainCar-v0 (optional) | Box(2) / Discrete(3) | Mean ≥ −110 over 100 episodes | Exploration stress test (vanilla agents *legitimately* fail — that's a finding, not a bug) |
+| **Pendulum-v1** (faithful port: g 10, m 1, l 1, dt 0.05, semi-implicit Euler, torque clip ±2, θ̇ clip ±8, no terminal, truncate@200) | Box(3) [cos θ, sin θ, θ̇] / **Box(1) torque ∈ [−2,2]** | Mean return ≥ −200 over 100 episodes | First **continuous-control** env; SAC flagship (random ≈ −1200) |
 
 Rush Hour note: board logic is reimplemented inside `MintPlayer.AI.ReinforcementLearning.Environments` (~150 LOC) rather
 than referencing `C:\Repos\Spelletjes\Rush Hour` (currently being modified by another
@@ -209,6 +213,8 @@ Per-game assignment:
 | **Snake — human play** | client-side only; a **JS timer** ticks the local TS engine, keyboard steers | — |
 | **MountainCar — watch-AI** | **B — server streams each tick `{position, velocity, action, reward, done}`** | `WS /api/mountaincar/live` |
 | **MountainCar — human play** | client-side only; a **JS timer** ticks the local TS physics, ←/→ push | — |
+| **Pendulum — watch-AI** | **B — server streams each tick `{cosTheta, sinTheta, angularVelocity, torque, reward, done}`** | `WS /api/pendulum/live` |
+| **Pendulum — human play** | client-side only; a **JS timer** ticks the local TS physics, ←/→ apply a continuous torque | — |
 
 Each new game ships **both modes**, which nicely demonstrates the two principles side by side:
 - **Watch-AI = principle B** — *backend* owns the loop + clock, streams frames, frontend is a pure renderer with

src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/ContinuousReplayBufferCheckpoint.cs

@@ -0,0 +1,65 @@
+using System.Text;
+using MintPlayer.AI.ReinforcementLearning.Core.Training;
+
+namespace MintPlayer.AI.ReinforcementLearning.Core.Checkpoints;
+
+/// <summary>
+/// Serializes a <see cref="ContinuousReplayBuffer"/> in the SDK's versioned binary form — the continuous
+/// counterpart of <see cref="ReplayBufferCheckpoint"/> (float action vectors, no mask). <see cref="Write"/>/
+/// <see cref="Read"/> emit a header-less payload for embedding in <see cref="SacTrainingState"/>; only the
+/// live prefix (entries 0..Count-1) is written, with <c>NextIndex</c> preserving the circular write position.
+/// </summary>
+public static class ContinuousReplayBufferCheckpoint
+{
+    public const string Kind = "continuous-replay-buffer";
+    private const int Version = 1;
+
+    /// <summary>Standalone, self-describing buffer file (header + payload).</summary>
+    public static void Save(ContinuousReplayBuffer buffer, Stream destination)
+    {
+        using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true);
+        CheckpointFormat.WriteHeader(writer, Kind, Version);
+        Write(buffer, writer);
+    }
+
+    /// <summary>Reads a buffer written by <see cref="Save"/>.</summary>
+    public static ContinuousReplayBuffer Load(Stream source)
+    {
+        using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true);
+        CheckpointFormat.ReadHeader(reader, Kind, Version);
+        return Read(reader);
+    }
+
+    /// <summary>Header-less payload, for embedding in a larger checkpoint stream.</summary>
+    public static void Write(ContinuousReplayBuffer buffer, BinaryWriter writer)
+    {
+        writer.Write(buffer.Capacity);
+        writer.Write(buffer.ObsDim);
+        writer.Write(buffer.ActionDim);
+        writer.Write(buffer.Count);
+        writer.Write(buffer.NextIndex);
+        CheckpointFormat.WriteFloats(writer, buffer.ObsData.AsSpan(0, buffer.Count * buffer.ObsDim));
+        CheckpointFormat.WriteFloats(writer, buffer.NextObsData.AsSpan(0, buffer.Count * buffer.ObsDim));
+        CheckpointFormat.WriteFloats(writer, buffer.ActionsData.AsSpan(0, buffer.Count * buffer.ActionDim));
+        CheckpointFormat.WriteFloats(writer, buffer.RewardsData.AsSpan(0, buffer.Count));
+        CheckpointFormat.WriteBools(writer, buffer.TerminatedData.AsSpan(0, buffer.Count));
+    }
+
+    /// <summary>Reads a header-less payload written by <see cref="Write"/>.</summary>
+    public static ContinuousReplayBuffer Read(BinaryReader reader)
+    {
+        int capacity = reader.ReadInt32();
+        int obsDim = reader.ReadInt32();
+        int actionDim = reader.ReadInt32();
+        int count = reader.ReadInt32();
+        int nextIndex = reader.ReadInt32();
+
+        var buffer = new ContinuousReplayBuffer(capacity, obsDim, actionDim) { Count = count, NextIndex = nextIndex };
+        CheckpointFormat.ReadFloats(reader).CopyTo(buffer.ObsData.AsSpan(0, count * obsDim));
+        CheckpointFormat.ReadFloats(reader).CopyTo(buffer.NextObsData.AsSpan(0, count * obsDim));
+        CheckpointFormat.ReadFloats(reader).CopyTo(buffer.ActionsData.AsSpan(0, count * actionDim));
+        CheckpointFormat.ReadFloats(reader).CopyTo(buffer.RewardsData.AsSpan(0, count));
+        CheckpointFormat.ReadBools(reader).CopyTo(buffer.TerminatedData.AsSpan(0, count));
+        return buffer;
+    }
+}