diff --git a/docs/ADDING_A_GAME.md b/docs/ADDING_A_GAME.md index 09db7ff..019d693 100644 --- a/docs/ADDING_A_GAME.md +++ b/docs/ADDING_A_GAME.md @@ -2,11 +2,18 @@ The mechanical, file-by-file checklist for adding a game to the playground, reverse-engineered from **Rush Hour** / **2048** / **Cube** (investigated 2026-06-15). Cross-refs: `PRD.md` §7 + §7.1 (interaction -models), `PLAN.md` M8–M10 (web slices) + M22 (MountainCar/Snake). +models), `PLAN.md` M8–M10 (web slices) + M22 (MountainCar/Snake) + M23 (Pendulum/SAC). For a new game `X` (lowercase env id `xgame`, PascalCase `XGame`), pick the **interaction principle** first (PRD §7.1): **A — compute-and-return** (HTTP, like Cube/2048/RushHour/Snake) or **B — live control stream** -(WebSocket, like MountainCar). Then work the six layers. +(WebSocket, like MountainCar/Pendulum). Then work the six layers. + +**Continuous-action games** (a real-valued action, like Pendulum) differ only at a few seams: the env is +`IEnvironment` with a **`BoxSpace` action space** (not `DiscreteSpace`), the agent is a +`ContinuousPolicyAgent` over a Gaussian policy net (greedy = tanh(mean), rescaled to the box bounds), trained with +**SAC** (`SacTrainer`), and the WS `EpisodeStreamer` is used with `TAct = float[]` (pass a zero-vector +`resetAction`). Serve-time checkpoint is the actor `Mlp` via `MlpCheckpoint` (critics/temperature are training-only). +Human play maps ←/→ to a continuous torque rather than a discrete button. ## Load-bearing conventions - **Model-store filename:** `FileModelStore.PathOf` maps `(environmentId, algorithmId)` → `/..ckpt`. diff --git a/docs/PLAN.md b/docs/PLAN.md index 86ea665..776795f 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -781,6 +781,40 @@ page with two modes (watch-AI WS client + human-play TS engine on a JS timer) + **Effort:** the first WS streamer is the new infra (~1 d, reused by both); then Snake ~1–2 d, MountainCar ~2–3 d (env + PPO). Land the reusable streamer + Snake first (simpler env), then MountainCar. +## M23 — Pendulum + SAC *(SDK breadth: first continuous control; designed 2026-06-15 by a 4-agent investigation, built same day)* + +The SDK's first **continuous-action** algorithm — every prior agent picks one of N discrete actions; SAC outputs a +real-valued torque. ✅ **BUILT + MEASURED 2026-06-15.** The four-agent investigation found the core is far more +ready than it looks: `IEnvironment`, `IAgent`, `Evaluator` are already generic over the action type, +`BoxSpace` is already a valid action space, and the autograd layer needs **zero new backend kernels**. + +**New Core machinery (the continuous pillar):** +- `Tensor.ConcatCols` / `SliceCols` — two small column-wise autograd ops (pure memory layout): `Concat` feeds the + critic its `concat(state, action)` input (routing the actor's gradient through the action half); `Slice` splits a + policy net's `[B, 2·A]` output into the mean and log-σ halves. +- `Nn/Normal.cs` — diagonal **squashed Gaussian** (the continuous mirror of `Categorical`): reparameterized + `tanh(mean + σ·ε)` sample, log-prob with the tanh change-of-variables correction, `Mode` = tanh(mean). `ε` is a + detached `Tensor.RandomNormal` leaf (stop-gradient for free); log-σ clamped to [−20, 2] before exp. +- `Training/ContinuousReplayBuffer.cs` (+ checkpoint) — sibling of `ReplayBuffer` with float-vector actions and no + mask, leaving the discrete buffer byte-compatible. +- `Training/SacTrainer.cs` — **SAC** (Haarnoja 2018): a squashed-Gaussian actor `[obs → 2·A]`, **twin Q-critics** + (plain `Mlp([obs+A,…,1])`, clipped double-Q targets), **Polyak soft target update** (a `SoftUpdate` helper, NOT + an overload of the hard `CopyFrom`), and an **auto-tuned entropy temperature** (log-α against target entropy −A). + `ContinuousPolicyAgent : IAgent` serves greedy = tanh(mean), rescaled to the box bounds. The + off-policy collection loop mirrors `DqnTrainer` (single env + replay, warmup, eval cadence, solve-threshold, + bitwise-resumable `SacTrainingState`). + +**PendulumEnv** (`Environments/PendulumEnv.cs`) — faithful Gymnasium Pendulum-v1: obs `[cos θ, sin θ, θ̇]` (Box 3), +action torque ∈ [−2,2] (**Box 1**), reward `−(θ² + 0.1·θ̇² + 0.001·τ²)` from the pre-update angle, **semi-implicit +Euler** (θ advances using the new θ̇), torque clamped (not thrown), **no terminal — truncate@200**. `IStatefulEnvironment`. + +**Measured:** SAC (`Hidden [128,128], 30k steps, ~6 min CPU`) reaches greedy eval **mean return −149.1 over 100 +episodes** (random ≈ −1200; past the −200 "solved" bar and the −150 solve-threshold). Bitwise-resume test green. +Seed shipped to `models/pendulum.sac.ckpt` (Git LFS). **Web (PRD §7.1):** `WS /api/pendulum/live` streams +`{cosTheta, sinTheta, angularVelocity, torque, reward, done}` (the `EpisodeStreamer` was generalized over `TAct`); +Angular `` rod + **watch-AI** (WS) / **swing-it-yourself** (client TS physics on a JS timer, ←/→ = continuous +torque). Serve-time checkpoint = actor-only via the existing `MlpCheckpoint` (critics/temperature are training-only). + ## Testing strategy (cross-cutting, from research) 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket. @@ -824,6 +858,8 @@ page with two modes (watch-AI WS client + human-play TS engine on a JS timer) + | Learning-curve levers P.7/P.8/P.9 (2026-06-14) | feed GPU, decouple pacing, per-update efficiency | parallel successor gen (GPU 0→95-100% util) + `--batch`; sample-paced curriculum + lighter eval; ε-loss target sync (freeze-proof) + `--lr`. Net: **~3,050 samples/s (~2× prior)**, GPU-bound at ~2 TFLOP/s | | General `IComputeBackend` port Phase 1 (2026-06-14) | every autograd op behind the seam; CPU bitwise-identical | all ops (Map/Zip/reductions/LogSoftmax/Gather/Huber/LayerNorm) routed through the backend; 224/224 green. Phase 2 (device-backed Tensor) parked far-future (no measured win at our scale) | | **M21 shortest-move solver — BWAS capability (2026-06-14, residual 1024×4, ~44k iters)** | **provably-optimal shallow; beat Kociemba's QTM mid-range** | batched A* (w=2.5, ≤40k exp, 12/depth): **12/12 & QTM-optimal through depth 10** (exactly d·qt), **100% solved through depth 12**, 83% d13, 75% d14; **every solve beats Kociemba's QTM, often ~2×** (d10 10 vs 19, d12 12.2 vs 29.3). Greedy (live curve) collapses ~d10-11 — search reads the net far deeper. Beats the earlier 1024×3 MLP at every deep level (d12 100% vs 80%). Full god's-number (26 QTM) NOT reached — honest two-tier story | +| M22 MountainCar (PPO) + Snake (masked Double+Dueling DQN) (2026-06-15) | MountainCar mean ≥ −110; Snake ≥ 5 food | MountainCar **−107.9, goal 100/100** (~120k steps, <1 min); Snake **22.1 food on 12×12** (trained on 6×6 → transfer) | +| M23 SAC continuous control (2026-06-15) | Pendulum mean ≥ −200 (gate median/3 ≥ −250) | **−149.1 over 100 eps** (Hidden [128,128], 30k steps, ~6 min CPU); random ≈ −1200; bitwise-resume green | ## Shipped (2026-06-11) — release engineering @@ -896,8 +932,9 @@ curriculum + lighter eval) and **✅ P.9** (LR scaling + ε-loss target sync). F 3. **AlphaZero-style fine-tune** — *started 2026-06-11, paused mid-campaign; see the "Fine-tune round" section under M11 for results so far and the resume command.* 3. **SDK breadth** (the demos exist to show range): algorithm coverage (✅ PPO action - masking *(done)*, ✅ dueling head *(done)*, maybe SAC), more envs (**MountainCar + Snake — designed, see M22**), - a TensorBoard writer, public-API stability/semver discipline, reproducibility guarantees. + masking *(done)*, ✅ dueling head *(done)*, ✅ SAC continuous control *(done, see M23)*), more envs + (✅ **MountainCar + Snake — M22**, ✅ **Pendulum continuous — M23**), a TensorBoard writer, public-API + stability/semver discipline, reproducibility guarantees. **"Switch algorithm, keep the work" — make the three reusable assets first-class.** When an algorithm plateaus, what carries over is the env (already portable via @@ -925,7 +962,7 @@ curriculum + lighter eval) and **✅ P.9** (LR scaling + ε-loss target sync). F Run the playground: `dotnet run --project src/RLDemo.Web` (Development spawns + proxies the Angular dev server itself — do not run `ng serve`). Console demos: -`dotnet run --project src/RLDemo.Console -c Release -- [grid|lake|cartpole|ppo|2048|2048dqn|rushhour|cube] +`dotnet run --project src/RLDemo.Console -c Release -- [grid|lake|cartpole|ppo|2048|2048dqn|rushhour|cube|pendulum] [seed] [--load] [--save] [--data ]`. Tests: `dotnet test` (`Category=Slow` for gates). Training campaigns (resume net + Adam + full training state from the model store): `dotnet run --project tools/MintPlayer.AI.ReinforcementLearning.Lab -c Release -- diff --git a/docs/PRD.md b/docs/PRD.md index bbf7d91..e1722c5 100644 --- a/docs/PRD.md +++ b/docs/PRD.md @@ -34,7 +34,8 @@ out (vectorized environments) without rewrites. 2. **Gymnasium-faithful environment API** for .NET — typed `Reset`/`Step` with separate `Terminated`/`Truncated`, spaces, seeding — so results are comparable to literature. 3. **Algorithm ladder**: tabular Q-learning/SARSA → REINFORCE → DQN (+Double/Dueling) → PPO - (vectorized envs, GAE). Each gated by a known-solved threshold before the next starts. + (vectorized envs, GAE) → SAC (off-policy continuous control). Each gated by a known-solved + threshold before the next starts. 4. **From-scratch numerics**: own tensor type, SIMD matmul, tape-based autograd, Adam — zero native dependencies in the core. 5. **Real games as benchmarks**: CartPole (literature-comparable flagship), **2048** and @@ -69,7 +70,9 @@ Explicitly out of scope to prevent the scope creep every research thread warned - Distributed training, ONNX export, model-based / offline RL - Unity/Godot adapters, netstandard multi-targeting *(NuGet publishing was originally out of scope here but shipped on 2026-06-11)* -- LunarLander or anything needing a physics engine (Box2D port ≈ a project in itself) +- LunarLander or anything needing a physics engine (Box2D port ≈ a project in itself). + *Classic-control continuous tasks (Pendulum) ARE in scope — their dynamics are a trivial ODE; + it's only the Box2D-class physics engines that stay out.* ## 4. Key decisions @@ -136,6 +139,7 @@ Pre-registered, objective "solved" definitions (fixed now so benchmarks stay mea | **2048** (4×4, spawn 90% 2 / 10% 4, invalid moves masked) | Box(16) log2-encoded / Discrete(4) | Pre-registered: reach 2048 tile in ≥ 10% of 100 eval games (DQN target); stretch: TD/n-tuple agent ≥ 80% | Owner's game #1; stochastic, showy | | **Rush Hour** (6×6 sliding-block; action = (vehicle, ±1 slide); reward −1/step, +100 exit; per-difficulty puzzle sets) | Box(72): vehicle-identity plane + red-car plane / Discrete(32 masked) | ≥ 90% of *easy* puzzle set solved within 2× optimal moves (optimal via built-in BFS solver) | Owner's game #2; sparse-reward planning, curriculum learning | | MountainCar-v0 (optional) | Box(2) / Discrete(3) | Mean ≥ −110 over 100 episodes | Exploration stress test (vanilla agents *legitimately* fail — that's a finding, not a bug) | +| **Pendulum-v1** (faithful port: g 10, m 1, l 1, dt 0.05, semi-implicit Euler, torque clip ±2, θ̇ clip ±8, no terminal, truncate@200) | Box(3) [cos θ, sin θ, θ̇] / **Box(1) torque ∈ [−2,2]** | Mean return ≥ −200 over 100 episodes | First **continuous-control** env; SAC flagship (random ≈ −1200) | Rush Hour note: board logic is reimplemented inside `MintPlayer.AI.ReinforcementLearning.Environments` (~150 LOC) rather than referencing `C:\Repos\Spelletjes\Rush Hour` (currently being modified by another @@ -209,6 +213,8 @@ Per-game assignment: | **Snake — human play** | client-side only; a **JS timer** ticks the local TS engine, keyboard steers | — | | **MountainCar — watch-AI** | **B — server streams each tick `{position, velocity, action, reward, done}`** | `WS /api/mountaincar/live` | | **MountainCar — human play** | client-side only; a **JS timer** ticks the local TS physics, ←/→ push | — | +| **Pendulum — watch-AI** | **B — server streams each tick `{cosTheta, sinTheta, angularVelocity, torque, reward, done}`** | `WS /api/pendulum/live` | +| **Pendulum — human play** | client-side only; a **JS timer** ticks the local TS physics, ←/→ apply a continuous torque | — | Each new game ships **both modes**, which nicely demonstrates the two principles side by side: - **Watch-AI = principle B** — *backend* owns the loop + clock, streams frames, frontend is a pure renderer with diff --git a/models/pendulum.sac.ckpt b/models/pendulum.sac.ckpt new file mode 100644 index 0000000..b258e74 --- /dev/null +++ b/models/pendulum.sac.ckpt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1426bf35cb67ef35ca23809d0e69636ea598fcfbfd8feec1cd30f53bcd56392b +size 69185 diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/ContinuousReplayBufferCheckpoint.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/ContinuousReplayBufferCheckpoint.cs new file mode 100644 index 0000000..f0e0e10 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/ContinuousReplayBufferCheckpoint.cs @@ -0,0 +1,65 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Training; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; + +/// +/// Serializes a in the SDK's versioned binary form — the continuous +/// counterpart of (float action vectors, no mask). / +/// emit a header-less payload for embedding in ; only the +/// live prefix (entries 0..Count-1) is written, with NextIndex preserving the circular write position. +/// +public static class ContinuousReplayBufferCheckpoint +{ + public const string Kind = "continuous-replay-buffer"; + private const int Version = 1; + + /// Standalone, self-describing buffer file (header + payload). + public static void Save(ContinuousReplayBuffer buffer, Stream destination) + { + using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.WriteHeader(writer, Kind, Version); + Write(buffer, writer); + } + + /// Reads a buffer written by . + public static ContinuousReplayBuffer Load(Stream source) + { + using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.ReadHeader(reader, Kind, Version); + return Read(reader); + } + + /// Header-less payload, for embedding in a larger checkpoint stream. + public static void Write(ContinuousReplayBuffer buffer, BinaryWriter writer) + { + writer.Write(buffer.Capacity); + writer.Write(buffer.ObsDim); + writer.Write(buffer.ActionDim); + writer.Write(buffer.Count); + writer.Write(buffer.NextIndex); + CheckpointFormat.WriteFloats(writer, buffer.ObsData.AsSpan(0, buffer.Count * buffer.ObsDim)); + CheckpointFormat.WriteFloats(writer, buffer.NextObsData.AsSpan(0, buffer.Count * buffer.ObsDim)); + CheckpointFormat.WriteFloats(writer, buffer.ActionsData.AsSpan(0, buffer.Count * buffer.ActionDim)); + CheckpointFormat.WriteFloats(writer, buffer.RewardsData.AsSpan(0, buffer.Count)); + CheckpointFormat.WriteBools(writer, buffer.TerminatedData.AsSpan(0, buffer.Count)); + } + + /// Reads a header-less payload written by . + public static ContinuousReplayBuffer Read(BinaryReader reader) + { + int capacity = reader.ReadInt32(); + int obsDim = reader.ReadInt32(); + int actionDim = reader.ReadInt32(); + int count = reader.ReadInt32(); + int nextIndex = reader.ReadInt32(); + + var buffer = new ContinuousReplayBuffer(capacity, obsDim, actionDim) { Count = count, NextIndex = nextIndex }; + CheckpointFormat.ReadFloats(reader).CopyTo(buffer.ObsData.AsSpan(0, count * obsDim)); + CheckpointFormat.ReadFloats(reader).CopyTo(buffer.NextObsData.AsSpan(0, count * obsDim)); + CheckpointFormat.ReadFloats(reader).CopyTo(buffer.ActionsData.AsSpan(0, count * actionDim)); + CheckpointFormat.ReadFloats(reader).CopyTo(buffer.RewardsData.AsSpan(0, count)); + CheckpointFormat.ReadBools(reader).CopyTo(buffer.TerminatedData.AsSpan(0, count)); + return buffer; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/SacTrainingState.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/SacTrainingState.cs new file mode 100644 index 0000000..fec7b31 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/SacTrainingState.cs @@ -0,0 +1,125 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Training; +using Tensor = MintPlayer.AI.ReinforcementLearning.Core.Numerics.Tensor; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; + +/// +/// The complete mutable state of a SAC training run — actor, twin critics + their targets, the actor and +/// (shared) critic optimizers, the entropy temperature (log-α) and its optimizer, the replay buffer, RNG +/// streams, current observation and (for envs) the +/// environment snapshot. Saving this and resuming via SacTrainer.Train(..., resume: state) continues +/// bitwise-identically to a run that was never interrupted. Mirrors . +/// +public sealed class SacTrainingState +{ + public const string Kind = "sac-state"; + private const int Version = 1; + + public required Mlp Actor { get; init; } + public required Mlp Critic1 { get; init; } + public required Mlp Critic2 { get; init; } + public required Mlp Target1 { get; init; } + public required Mlp Target2 { get; init; } + + public required Adam ActorOptimizer { get; init; } + public required Adam CriticOptimizer { get; init; } // bound to Critic1.Parameters() ++ Critic2.Parameters() + + /// Entropy temperature in log-space: a 1-element grad-carrying leaf. α = exp(LogAlpha). + public required Tensor LogAlpha { get; init; } + /// Optimizer for ; null when the temperature is fixed. + public Adam? AlphaOptimizer { get; init; } + + public required ContinuousReplayBuffer Buffer { get; init; } + public required Xoshiro256StarStar PolicyRng { get; init; } + public required Xoshiro256StarStar BufferRng { get; init; } + + public float[] CurrentObs { get; set; } = []; + public int StepsCompleted { get; set; } + public float LastCriticLoss { get; set; } + public float LastActorLoss { get; set; } + public double LastEval { get; set; } = double.NegativeInfinity; + + /// Opaque environment snapshot; null when the env is not IStatefulEnvironment. + public byte[]? EnvState { get; set; } + + public void Save(Stream destination) + { + using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.WriteHeader(writer, Kind, Version); + + MlpCheckpoint.Write(Actor, writer); + MlpCheckpoint.Write(Critic1, writer); + MlpCheckpoint.Write(Critic2, writer); + MlpCheckpoint.Write(Target1, writer); + MlpCheckpoint.Write(Target2, writer); + + AdamCheckpoint.Write(ActorOptimizer, writer); + AdamCheckpoint.Write(CriticOptimizer, writer); + + writer.Write(AlphaOptimizer is not null); + writer.Write(LogAlpha.Data[0]); + if (AlphaOptimizer is not null) AdamCheckpoint.Write(AlphaOptimizer, writer); + + ContinuousReplayBufferCheckpoint.Write(Buffer, writer); + CheckpointFormat.WriteRngState(writer, PolicyRng); + CheckpointFormat.WriteRngState(writer, BufferRng); + CheckpointFormat.WriteFloats(writer, CurrentObs); + writer.Write(StepsCompleted); + writer.Write(LastCriticLoss); + writer.Write(LastActorLoss); + writer.Write(LastEval); + + int envStateLength = EnvState?.Length ?? -1; + writer.Write(envStateLength); + if (EnvState is not null) writer.Write(EnvState); + } + + public static SacTrainingState Load(Stream source) + { + using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.ReadHeader(reader, Kind, Version); + + var actor = MlpCheckpoint.Read(reader); + var critic1 = MlpCheckpoint.Read(reader); + var critic2 = MlpCheckpoint.Read(reader); + var target1 = MlpCheckpoint.Read(reader); + var target2 = MlpCheckpoint.Read(reader); + + var actorOptimizer = AdamCheckpoint.Read(actor.Parameters(), reader); + var criticOptimizer = AdamCheckpoint.Read([.. critic1.Parameters(), .. critic2.Parameters()], reader); + + bool autoAlpha = reader.ReadBoolean(); + var logAlpha = new Tensor([reader.ReadSingle()], 1) { RequiresGrad = true }; + Adam? alphaOptimizer = autoAlpha ? AdamCheckpoint.Read([logAlpha], reader) : null; + + var buffer = ContinuousReplayBufferCheckpoint.Read(reader); + + var state = new SacTrainingState + { + Actor = actor, + Critic1 = critic1, + Critic2 = critic2, + Target1 = target1, + Target2 = target2, + ActorOptimizer = actorOptimizer, + CriticOptimizer = criticOptimizer, + LogAlpha = logAlpha, + AlphaOptimizer = alphaOptimizer, + Buffer = buffer, + PolicyRng = CheckpointFormat.ReadRngState(reader), + BufferRng = CheckpointFormat.ReadRngState(reader), + CurrentObs = CheckpointFormat.ReadFloats(reader), + StepsCompleted = reader.ReadInt32(), + LastCriticLoss = reader.ReadSingle(), + LastActorLoss = reader.ReadSingle(), + LastEval = reader.ReadDouble(), + }; + + int envStateLength = reader.ReadInt32(); + if (envStateLength >= 0) state.EnvState = reader.ReadBytes(envStateLength); + return state; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/Normal.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/Normal.cs new file mode 100644 index 0000000..a5ee86b --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/Normal.cs @@ -0,0 +1,68 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Nn; + +/// +/// Diagonal Gaussian over continuous actions parameterized by per-action mean and log-σ [B,A], with the +/// tanh squash SAC uses to bound actions to (−1,1). Reparameterized samples and their log-probs are built +/// from autograd ops so the actor loss differentiates through the sample; the N(0,1) noise is a detached +/// constant (a bare has no graph parents, so gradient never leaks into it). The mirror +/// of for continuous control. +/// +public readonly struct Normal +{ + /// log-σ clamp range (Haarnoja SAC convention) — keeps σ = exp(log-σ) from over/underflowing. + public const float LogStdMin = -20f; + public const float LogStdMax = 2f; + + private static readonly float HalfLog2Pi = 0.5f * MathF.Log(2f * MathF.PI); + + private readonly Tensor _mean; // [B,A] + private readonly Tensor _logStd; // [B,A], already clamped + private readonly Tensor _std; // exp(log-σ), cached + + public Normal(Tensor mean, Tensor logStd) + { + _mean = mean; + _logStd = logStd; + _std = logStd.Exp(); + } + + /// + /// Builds a squashed Gaussian from a policy net's [B,2·A] output: the first A columns are the mean, the + /// last A the log-σ (clamped to ..). + /// + public static Normal FromNetOutput(Tensor netOutput, int actionDim) + { + var mean = netOutput.SliceCols(0, actionDim); + var logStd = netOutput.SliceCols(actionDim, actionDim).Clamp(LogStdMin, LogStdMax); + return new Normal(mean, logStd); + } + + /// + /// Reparameterized squashed sample: a = tanh(mean + σ·ε), ε ~ N(0,1) detached. Returns the action [B,A] + /// in (−1,1) and its log-prob [B] (the Gaussian log-prob of the pre-squash sample minus the tanh + /// change-of-variables correction, summed over action dims). + /// + public (Tensor Action, Tensor LogProb) RSample(Xoshiro256StarStar rng) + { + var eps = Tensor.RandomNormal(rng, 0f, 1f, _mean.Shape); // constant leaf — stop-gradient + var preSquash = _mean.Add(_std.Mul(eps)); // mean + σ·ε + var action = preSquash.Tanh(); // squashed to (−1,1) + return (action, LogProb(eps, action)); + } + + /// Deterministic action for evaluation/serving: tanh(mean) → [B,A]. + public Tensor Mode() => _mean.Tanh(); + + // log N(a;μ,σ) per dim, with (a−μ)/σ ≡ ε, minus the tanh correction −log(1−tanh²+ε); summed over dims. + private Tensor LogProb(Tensor eps, Tensor action) + { + var gaussian = _logStd.MulScalar(-1f) + .Sub(eps.Square().MulScalar(0.5f)) + .Sub(Tensor.Full(HalfLog2Pi, _mean.Shape)); + var correction = Tensor.Full(1f + 1e-6f, action.Shape).Sub(action.Square()).Log(); + return gaussian.Sub(correction).SumRows(); + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorOps.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorOps.cs index 2a698d6..1bde128 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorOps.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Numerics/TensorOps.cs @@ -316,6 +316,69 @@ public Tensor Reshape(params int[] shape) }); } + /// + /// Column-wise concatenation of two row-aligned tensors: [B,n₁] ⊕ [B,n₂] → [B,n₁+n₂]. Gradient + /// routes the left columns back to this and the right columns to . + /// (SAC's critic takes concat(state, action), and the action half must carry the actor's gradient.) + /// Pure memory layout, so it copies directly rather than dispatching to the compute backend. + /// + public Tensor ConcatCols(Tensor other) + { + CheckRank2(this); CheckRank2(other); + if (Rows != other.Rows) + throw new ArgumentException($"ConcatCols row mismatch: {Rows} vs {other.Rows}."); + int rows = Rows, left = Cols, right = other.Cols, total = left + right; + + var data = new float[rows * total]; + for (int r = 0; r < rows; r++) + { + Array.Copy(Data, r * left, data, r * total, left); + Array.Copy(other.Data, r * right, data, r * total + left, right); + } + + return MakeResult(data, [rows, total], [this, other], result => () => + { + var dy = result.Grad!; + if (NeedsGrad) + { + EnsureGrad(); + for (int r = 0; r < rows; r++) + for (int c = 0; c < left; c++) Grad![r * left + c] += dy[r * total + c]; + } + if (other.NeedsGrad) + { + other.EnsureGrad(); + for (int r = 0; r < rows; r++) + for (int c = 0; c < right; c++) other.Grad![r * right + c] += dy[r * total + left + c]; + } + }); + } + + /// + /// Extracts a contiguous column range: [B,N] → [B,] starting at + /// . Gradient scatters back into the source columns. (Splits a policy net's + /// [B,2·A] output into the mean and log-σ halves of a Gaussian.) Pure layout op. + /// + public Tensor SliceCols(int start, int count) + { + CheckRank2(this); + if (start < 0 || count < 0 || start + count > Cols) + throw new ArgumentOutOfRangeException(nameof(start), $"Slice [{start},{start + count}) out of [0,{Cols})."); + int rows = Rows, cols = Cols; + + var data = new float[rows * count]; + for (int r = 0; r < rows; r++) + Array.Copy(Data, r * cols + start, data, r * count, count); + + return MakeResult(data, [rows, count], [this], result => () => + { + EnsureGrad(); + var dy = result.Grad!; + for (int r = 0; r < rows; r++) + for (int c = 0; c < count; c++) Grad![r * cols + start + c] += dy[r * count + c]; + }); + } + private static void AccumulateGrad(Tensor tensor, float[] grad) { if (!tensor.NeedsGrad) return; diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/ContinuousReplayBuffer.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/ContinuousReplayBuffer.cs new file mode 100644 index 0000000..3108fbd --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/ContinuousReplayBuffer.cs @@ -0,0 +1,87 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Training; + +/// +/// Circular experience-replay buffer for off-policy continuous-action learning (SAC). The sibling of +/// : it stores a real-valued action vector per transition instead of a discrete +/// index, and carries no action mask (continuous policies aren't masked). Like the discrete buffer it stores +/// the terminated flag ONLY — a time-limit truncated transition must still bootstrap from its +/// next state (conflating the two is the classic silent value-capping bug, PRD §8). +/// +public sealed class ContinuousReplayBuffer +{ + private readonly int _obsDim; + private readonly int _actionDim; + private readonly float[] _obs; + private readonly float[] _nextObs; + private readonly float[] _actions; + private readonly float[] _rewards; + private readonly bool[] _terminated; + private int _next; + + public ContinuousReplayBuffer(int capacity, int obsDim, int actionDim) + { + Capacity = capacity; + _obsDim = obsDim; + _actionDim = actionDim; + _obs = new float[capacity * obsDim]; + _nextObs = new float[capacity * obsDim]; + _actions = new float[capacity * actionDim]; + _rewards = new float[capacity]; + _terminated = new bool[capacity]; + } + + public int Capacity { get; } + public int Count { get; internal set; } + + // Internal state access for checkpointing (Checkpoints/ContinuousReplayBufferCheckpoint.cs). + internal int ObsDim => _obsDim; + internal int ActionDim => _actionDim; + internal float[] ObsData => _obs; + internal float[] NextObsData => _nextObs; + internal float[] ActionsData => _actions; + internal float[] RewardsData => _rewards; + internal bool[] TerminatedData => _terminated; + internal int NextIndex { get => _next; set => _next = value; } + + public void Add(ReadOnlySpan obs, ReadOnlySpan action, double reward, + ReadOnlySpan nextObs, bool terminated) + { + obs.CopyTo(_obs.AsSpan(_next * _obsDim, _obsDim)); + nextObs.CopyTo(_nextObs.AsSpan(_next * _obsDim, _obsDim)); + action.CopyTo(_actions.AsSpan(_next * _actionDim, _actionDim)); + _rewards[_next] = (float)reward; + _terminated[_next] = terminated; + + _next = (_next + 1) % Capacity; + Count = Math.Min(Count + 1, Capacity); + } + + public Batch Sample(int batchSize, Xoshiro256StarStar rng) + { + var batch = new Batch(batchSize, _obsDim, _actionDim); + for (int i = 0; i < batchSize; i++) + { + int index = rng.NextInt(Count); + _obs.AsSpan(index * _obsDim, _obsDim).CopyTo(batch.Obs.AsSpan(i * _obsDim, _obsDim)); + _nextObs.AsSpan(index * _obsDim, _obsDim).CopyTo(batch.NextObs.AsSpan(i * _obsDim, _obsDim)); + _actions.AsSpan(index * _actionDim, _actionDim).CopyTo(batch.Actions.AsSpan(i * _actionDim, _actionDim)); + batch.Rewards[i] = _rewards[index]; + batch.Terminated[i] = _terminated[index]; + } + return batch; + } + + public sealed class Batch(int size, int obsDim, int actionDim) + { + public int Size { get; } = size; + public int ObsDim { get; } = obsDim; + public int ActionDim { get; } = actionDim; + public float[] Obs { get; } = new float[size * obsDim]; + public float[] NextObs { get; } = new float[size * obsDim]; + public float[] Actions { get; } = new float[size * actionDim]; + public float[] Rewards { get; } = new float[size]; + public bool[] Terminated { get; } = new bool[size]; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Training/SacTrainer.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/SacTrainer.cs new file mode 100644 index 0000000..330557a --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Training/SacTrainer.cs @@ -0,0 +1,296 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Agents; +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Environments; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Training; + +public sealed record SacOptions +{ + public int[] Hidden { get; init; } = [256, 256]; + public double Gamma { get; init; } = 0.99; + public float LearningRate { get; init; } = 3e-4f; + + /// Polyak coefficient for the soft target-critic update θ′ ← τθ + (1−τ)θ′, applied every step. + public float Tau { get; init; } = 0.005f; + + public int BufferCapacity { get; init; } = 1_000_000; + public int BatchSize { get; init; } = 256; + + /// Uniform-random exploration steps before learning starts (and before the policy is queried). + public int WarmupSteps { get; init; } = 1_000; + public int TrainEvery { get; init; } = 1; + + /// Critic+actor updates performed per environment step (SAC update-to-data ratio). + public int GradientSteps { get; init; } = 1; + + public int MaxSteps { get; init; } = 100_000; + public float MaxGradNorm { get; init; } = 10f; + + /// Auto-tune the entropy temperature against a target entropy (Haarnoja 2018b). Recommended. + public bool AutoTuneAlpha { get; init; } = true; + /// Initial temperature; seeds log-α when auto-tuning, or fixes α when not. + public float InitialAlpha { get; init; } = 0.2f; + /// Target entropy for the α objective; defaults to −actionDim (the SAC heuristic) when null. + public float? TargetEntropy { get; init; } + + public int EvalEvery { get; init; } = 5_000; + public int EvalEpisodes { get; init; } = 20; + + /// Training stops early once a deterministic evaluation reaches this mean return. + public double? SolveThreshold { get; init; } + + public Action? OnProgress { get; init; } +} + +public readonly record struct SacProgress( + int Step, int MaxSteps, double EvalMeanReturn, float Alpha, float CriticLoss, float ActorLoss); + +public sealed record SacResult( + ContinuousPolicyAgent Agent, Mlp Actor, Mlp Critic1, Mlp Critic2, + int StepsTrained, double FinalEvalReturn, SacTrainingState State); + +/// +/// Acts from a squashed-Gaussian actor (SAC). Greedy = tanh(mean) (deterministic); stochastic = a +/// reparameterized squashed sample. The native (−1,1) action is rescaled to the environment's +/// bounds, so the agent plugs straight into and live demos. +/// +public sealed class ContinuousPolicyAgent(Mlp actor, int actionDim, float[] low, float[] high, Xoshiro256StarStar rng) + : IAgent +{ + public float[] Act(float[] observation, bool greedy = false) + { + using (GradMode.NoGrad()) + { + var netOutput = actor.Forward(new Tensor(observation, 1, observation.Length)); + float[] native = greedy + ? netOutput.SliceCols(0, actionDim).Tanh().Data + : Normal.FromNetOutput(netOutput, actionDim).RSample(rng).Action.Data; + return SacTrainer.ScaleToBounds(native, low, high); + } + } +} + +/// +/// Soft Actor-Critic (Haarnoja et al. 2018) — off-policy, maximum-entropy continuous control. A +/// squashed-Gaussian actor, twin Q-critics with clipped double-Q targets, soft (Polyak) target updates and +/// an auto-tuned entropy temperature. The off-policy collection loop mirrors +/// (single env + replay, warmup, periodic eval, solve-threshold early-out); only the update differs. +/// Single-file reference implementation per the CleanRL discipline (PLAN M23). +/// +public static class SacTrainer +{ + /// + /// A previously saved training state to continue from. The same options and master seed must be passed + /// as in the original run; on envs the resumed run is bitwise-identical + /// to one that was never interrupted. + /// + public static SacResult Train(IEnvironment env, SacOptions options, SeedSequence seeds, + SacTrainingState? resume = null) + { + int obsDim = ((BoxSpace)env.ObservationSpace).Dimensions; + var actionBox = (BoxSpace)env.ActionSpace; + int actionDim = actionBox.Dimensions; + float[] low = actionBox.Low, high = actionBox.High; + float targetEntropy = options.TargetEntropy ?? -actionDim; + + SacTrainingState state; + float[] obs; + if (resume is null) + { + var initRng = seeds.CreateRng(RngStreams.Init); + var actor = new Mlp([obsDim, .. options.Hidden, 2 * actionDim], initRng, Activation.Relu); + Mlp MakeCritic() => new([obsDim + actionDim, .. options.Hidden, 1], initRng, Activation.Relu); + var critic1 = MakeCritic(); + var critic2 = MakeCritic(); + var target1 = MakeCritic(); + var target2 = MakeCritic(); + target1.CopyFrom(critic1); + target2.CopyFrom(critic2); + + var logAlpha = new Tensor([MathF.Log(options.InitialAlpha)], 1) { RequiresGrad = true }; + state = new SacTrainingState + { + Actor = actor, + Critic1 = critic1, + Critic2 = critic2, + Target1 = target1, + Target2 = target2, + ActorOptimizer = new Adam(actor.Parameters(), options.LearningRate), + CriticOptimizer = new Adam([.. critic1.Parameters(), .. critic2.Parameters()], options.LearningRate), + LogAlpha = logAlpha, + AlphaOptimizer = options.AutoTuneAlpha ? new Adam([logAlpha], options.LearningRate) : null, + Buffer = new ContinuousReplayBuffer(options.BufferCapacity, obsDim, actionDim), + PolicyRng = seeds.CreateRng(RngStreams.Policy), + BufferRng = seeds.CreateRng(RngStreams.Buffer), + }; + (obs, _) = env.Reset(seeds.Derive(RngStreams.Environment)); + } + else + { + state = resume; + if (state.Buffer.Capacity != options.BufferCapacity) + throw new ArgumentException($"Resume state buffer capacity {state.Buffer.Capacity} != options {options.BufferCapacity}."); + if (state.CurrentObs.Length != obsDim) + throw new ArgumentException($"Resume state obs dim {state.CurrentObs.Length} != environment {obsDim}."); + + if (state.EnvState is not null && env is IStatefulEnvironment stateful) + { + stateful.RestoreState(state.EnvState); + obs = state.CurrentObs; + } + else + { + (obs, _) = env.Reset(seeds.Derive(RngStreams.Environment)); + } + } + + var agent = new ContinuousPolicyAgent(state.Actor, actionDim, low, high, state.PolicyRng); + float lastCriticLoss = state.LastCriticLoss, lastActorLoss = state.LastActorLoss; + double lastEval = state.LastEval; + + SacResult Finish(int stepsTrained) + { + state.CurrentObs = obs; + state.StepsCompleted = stepsTrained; + state.LastCriticLoss = lastCriticLoss; + state.LastActorLoss = lastActorLoss; + state.LastEval = lastEval; + state.EnvState = (env as IStatefulEnvironment)?.SaveState(); + return new SacResult(agent, state.Actor, state.Critic1, state.Critic2, stepsTrained, lastEval, state); + } + + for (int step = state.StepsCompleted + 1; step <= options.MaxSteps; step++) + { + // Native (−1,1) action: uniform random during warmup, else a stochastic policy sample. + float[] nativeAction = state.Buffer.Count < options.WarmupSteps + ? RandomNative(actionDim, state.PolicyRng) + : SamplePolicyNative(state.Actor, obs, actionDim, state.PolicyRng); + var result = env.Step(ScaleToBounds(nativeAction, low, high)); + + state.Buffer.Add(obs, nativeAction, result.Reward, result.Observation, result.Terminated); + obs = result.Done ? env.Reset().Observation : result.Observation; + + if (state.Buffer.Count >= options.WarmupSteps && state.Buffer.Count >= options.BatchSize + && step % options.TrainEvery == 0) + for (int g = 0; g < options.GradientSteps; g++) + (lastCriticLoss, lastActorLoss) = TrainStep(state, options, actionDim, targetEntropy); + + if (step % options.EvalEvery == 0) + { + lastEval = Evaluator.Evaluate(env, agent, options.EvalEpisodes, seeds.Derive(RngStreams.Evaluation)).MeanReturn; + options.OnProgress?.Invoke(new SacProgress( + step, options.MaxSteps, lastEval, MathF.Exp(state.LogAlpha.Data[0]), lastCriticLoss, lastActorLoss)); + (obs, _) = env.Reset(); // evaluation ended mid-episode; start fresh + + if (options.SolveThreshold.HasValue && lastEval >= options.SolveThreshold.Value) + return Finish(step); + } + } + + return Finish(options.MaxSteps); + } + + private static (float CriticLoss, float ActorLoss) TrainStep( + SacTrainingState state, SacOptions options, int actionDim, float targetEntropy) + { + var batch = state.Buffer.Sample(options.BatchSize, state.BufferRng); + int b = batch.Size; + float alpha = MathF.Exp(state.LogAlpha.Data[0]); + + var sObs = new Tensor(batch.Obs, b, batch.ObsDim); + var sAction = new Tensor(batch.Actions, b, actionDim); + var sNext = new Tensor(batch.NextObs, b, batch.ObsDim); + + // Critic target y = r + γ(1−terminated)·(min(Q1′,Q2′)(s′,a′) − α·log π(a′|s′)), gradient-free. + var targets = new float[b]; + using (GradMode.NoGrad()) + { + var (nextAction, nextLogProb) = Normal.FromNetOutput(state.Actor.Forward(sNext), actionDim).RSample(state.PolicyRng); + var saNext = sNext.ConcatCols(nextAction); + var q1Next = state.Target1.Forward(saNext); + var q2Next = state.Target2.Forward(saNext); + for (int i = 0; i < b; i++) + { + double minQ = Math.Min(q1Next.Data[i], q2Next.Data[i]); + double soft = minQ - alpha * nextLogProb.Data[i]; + targets[i] = (float)(batch.Rewards[i] + (batch.Terminated[i] ? 0 : options.Gamma * soft)); + } + } + var yTarget = new Tensor(targets, b); + + // Critic update: MSE(Q1,y) + MSE(Q2,y) over both critics in one step. + state.CriticOptimizer.ZeroGrad(); + var sa = sObs.ConcatCols(sAction); + var q1 = state.Critic1.Forward(sa).Reshape(b); + var q2 = state.Critic2.Forward(sa).Reshape(b); + var criticLoss = q1.MseLoss(yTarget).Add(q2.MseLoss(yTarget)); + criticLoss.Backward(); + state.CriticOptimizer.ClipGradNorm(options.MaxGradNorm); + state.CriticOptimizer.Step(); + + // Actor update: E[α·log π(a|s) − min(Q1,Q2)(s, a_reparam)]. + state.ActorOptimizer.ZeroGrad(); + var (action, logProb) = Normal.FromNetOutput(state.Actor.Forward(sObs), actionDim).RSample(state.PolicyRng); + var saPi = sObs.ConcatCols(action); + var minQpi = state.Critic1.Forward(saPi).Reshape(b).Min(state.Critic2.Forward(saPi).Reshape(b)); + var actorLoss = logProb.MulScalar(alpha).Sub(minQpi).Mean(); + actorLoss.Backward(); + state.ActorOptimizer.ClipGradNorm(options.MaxGradNorm); + state.ActorOptimizer.Step(); + + // Temperature update: −E[log α · (log π(a|s) + targetEntropy)], with log π treated as constant. + if (state.AlphaOptimizer is not null) + { + float coef = 0f; + for (int i = 0; i < b; i++) coef += logProb.Data[i] + targetEntropy; + coef /= b; + state.AlphaOptimizer.ZeroGrad(); + state.LogAlpha.MulScalar(-coef).Backward(); + state.AlphaOptimizer.Step(); + } + + SoftUpdate(state.Target1, state.Critic1, options.Tau); + SoftUpdate(state.Target2, state.Critic2, options.Tau); + + return (criticLoss.Data[0], actorLoss.Data[0]); + } + + /// Polyak soft update of a target net toward a source net: θ′ ← τθ + (1−τ)θ′. + private static void SoftUpdate(IModule target, IModule source, float tau) + { + using var t = target.Parameters().GetEnumerator(); + using var s = source.Parameters().GetEnumerator(); + while (t.MoveNext() && s.MoveNext()) + { + var td = t.Current.Data; + var sd = s.Current.Data; + for (int i = 0; i < td.Length; i++) + td[i] = tau * sd[i] + (1f - tau) * td[i]; + } + } + + private static float[] RandomNative(int actionDim, Xoshiro256StarStar rng) + { + var a = new float[actionDim]; + for (int i = 0; i < actionDim; i++) a[i] = (float)(2.0 * rng.NextDouble() - 1.0); + return a; + } + + private static float[] SamplePolicyNative(Mlp actor, float[] obs, int actionDim, Xoshiro256StarStar rng) + { + using (GradMode.NoGrad()) + return Normal.FromNetOutput(actor.Forward(new Tensor(obs, 1, obs.Length)), actionDim).RSample(rng).Action.Data; + } + + /// Maps a native action in [−1,1]^k to the environment's per-dimension box bounds. + public static float[] ScaleToBounds(float[] native, float[] low, float[] high) + { + var scaled = new float[native.Length]; + for (int i = 0; i < native.Length; i++) + scaled[i] = low[i] + (native[i] + 1f) * 0.5f * (high[i] - low[i]); + return scaled; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/PendulumEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/PendulumEnv.cs new file mode 100644 index 0000000..d4de0b6 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/PendulumEnv.cs @@ -0,0 +1,137 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Environments; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Environments; + +/// +/// Faithful port of Gymnasium Pendulum-v1 (classic_control/pendulum.py): swing a frictionless rod upright +/// and hold it there by applying a bounded torque. The SDK's first continuous-action environment — +/// observation [cos θ, sin θ, θ̇] (Box, 3-dim), action a single torque ∈ [−2, 2] (Box, 1-dim). Reward is the +/// negative cost −(θ² + 0.1·θ̇² + 0.001·τ²) measured from the angle BEFORE the integration step (0 upright, +/// at rest, no torque is the best attainable). There is no terminal state — every episode is truncated at the +/// 200-step cap. A competent SAC policy reaches a mean return around −150; random ≈ −1200. +/// +public sealed class PendulumEnv : IEnvironment, IStatefulEnvironment +{ + public const double Gravity = 10.0; + public const double Mass = 1.0; + public const double Length = 1.0; + public const double Dt = 0.05; + public const double MaxTorque = 2.0; + public const double MaxSpeed = 8.0; + public const int DefaultMaxEpisodeSteps = 200; + + private readonly int _maxEpisodeSteps; + + private Xoshiro256StarStar _rng = new(0); + private double _theta, _thetaDot; + private int _elapsedSteps; + private bool _done = true; + + public PendulumEnv(int maxEpisodeSteps = DefaultMaxEpisodeSteps) + { + _maxEpisodeSteps = maxEpisodeSteps; + // Observation is already well-scaled: cos/sin ∈ [-1,1] and θ̇ ∈ [-8,8], so (unlike MountainCar) no + // extra normalization is needed for a dense net to see every component. + ObservationSpace = new BoxSpace([-1f, -1f, (float)-MaxSpeed], [1f, 1f, (float)MaxSpeed]); + ActionSpace = new BoxSpace((float)-MaxTorque, (float)MaxTorque, 1); + } + + public Space ObservationSpace { get; } + public Space ActionSpace { get; } + + public double Theta => _theta; + public double AngularVelocity => _thetaDot; + + public (float[] Observation, EnvInfo Info) Reset(ulong? seed = null) + { + if (seed.HasValue) + _rng = new Xoshiro256StarStar(seed.Value); + _theta = _rng.NextDouble(-Math.PI, Math.PI); + _thetaDot = _rng.NextDouble(-1.0, 1.0); + _elapsedSteps = 0; + _done = false; + return (Observation(), EnvInfo.Empty); + } + + public StepResult Step(float[] action) + { + if (_done) + throw new InvalidOperationException("Episode is done; call Reset() before stepping."); + + // Continuous control clamps out-of-range torque rather than throwing (Gym clips; "define errors out + // of existence" — a policy whose σ pushes past the bound shouldn't crash the env). + double torque = Math.Clamp(action.Length > 0 ? action[0] : 0.0, -MaxTorque, MaxTorque); + + // Cost uses the angle BEFORE integration, normalized to (−π, π]. + double normTheta = NormalizeAngle(_theta); + double cost = normTheta * normTheta + 0.1 * _thetaDot * _thetaDot + 0.001 * torque * torque; + + // Gymnasium semi-implicit Euler: θ advances using the NEW angular velocity. + double newThetaDot = _thetaDot + (3.0 * Gravity / (2.0 * Length) * Math.Sin(_theta) + + 3.0 / (Mass * Length * Length) * torque) * Dt; + newThetaDot = Math.Clamp(newThetaDot, -MaxSpeed, MaxSpeed); + _theta += newThetaDot * Dt; + _thetaDot = newThetaDot; + _elapsedSteps++; + + bool truncated = _elapsedSteps >= _maxEpisodeSteps; + _done = truncated; + return new StepResult(Observation(), -cost, false, truncated, EnvInfo.Empty); + } + + /// Pins the physics state directly (golden-trajectory tests, demos). + public void SetState(double theta, double thetaDot) + { + (_theta, _thetaDot) = (theta, thetaDot); + _elapsedSteps = 0; + _done = false; + } + + private float[] Observation() => [(float)Math.Cos(_theta), (float)Math.Sin(_theta), (float)_thetaDot]; + + // Wrap to (−π, π], matching Gym's angle_normalize. + private static double NormalizeAngle(double theta) + { + double wrapped = (theta + Math.PI) % (2.0 * Math.PI); + if (wrapped < 0) wrapped += 2.0 * Math.PI; + return wrapped - Math.PI; + } + + public byte[] SaveState() + { + using var stream = new MemoryStream(); + using var writer = new BinaryWriter(stream); + var (s0, s1, s2, s3) = _rng.GetState(); + writer.Write(s0); writer.Write(s1); writer.Write(s2); writer.Write(s3); + writer.Write(_theta); + writer.Write(_thetaDot); + writer.Write(_elapsedSteps); + writer.Write(_done); + writer.Flush(); + return stream.ToArray(); + } + + public void RestoreState(byte[] state) + { + using var reader = new BinaryReader(new MemoryStream(state)); + _rng.SetState(reader.ReadUInt64(), reader.ReadUInt64(), reader.ReadUInt64(), reader.ReadUInt64()); + _theta = reader.ReadDouble(); + _thetaDot = reader.ReadDouble(); + _elapsedSteps = reader.ReadInt32(); + _done = reader.ReadBoolean(); + } + + public string RenderString() + { + // A clock-face glyph: the rod points toward the current angle (θ=0 is straight up). + const int width = 21; + double x = Math.Sin(_theta), y = Math.Cos(_theta); + int col = (int)Math.Round((x + 1) / 2 * (width - 1)); + var sb = new StringBuilder(); + sb.Append(y >= 0 ? "up " : "down "); + for (int i = 0; i < width; i++) sb.Append(i == col ? 'O' : '-'); + return sb.ToString(); + } +} diff --git a/src/RLDemo.Console/Program.cs b/src/RLDemo.Console/Program.cs index 7455da0..0e2de63 100644 --- a/src/RLDemo.Console/Program.cs +++ b/src/RLDemo.Console/Program.cs @@ -12,10 +12,10 @@ using MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; using MintPlayer.AI.ReinforcementLearning.Environments.RushHour; -// Usage: RLDemo.Console [grid|lake|cartpole|ppo|2048|2048dqn|rushhour|cube ...] [seed] [--load] [--save] [--data ] -// no env args = run everything except 2048dqn (DQN needs a long budget there). +// Usage: RLDemo.Console [grid|lake|cartpole|ppo|2048|2048dqn|rushhour|cube|pendulum ...] [seed] [--load] [--save] [--data ] +// no env args = run everything except 2048dqn and pendulum (both need a long budget). // --load: skip training when the model store has a checkpoint; --save: checkpoint after training. -string[] knownSections = ["grid", "lake", "cartpole", "ppo", "2048", "2048dqn", "rushhour", "cube"]; +string[] knownSections = ["grid", "lake", "cartpole", "ppo", "2048", "2048dqn", "rushhour", "cube", "pendulum"]; var selected = new HashSet(StringComparer.OrdinalIgnoreCase); ulong masterSeed = 42; bool loadModels = false, saveModels = false; @@ -31,7 +31,7 @@ else Console.WriteLine($"(ignoring unknown argument '{arg}')"); } bool ShouldRun(string name) => selected.Count == 0 - ? name != "2048dqn" + ? name is not ("2048dqn" or "pendulum") : selected.Contains(name); bool animate = !Console.IsOutputRedirected; var store = new FileModelStore(dataDir); @@ -403,9 +403,72 @@ void SaveMlp(string envId, string algoId, Mlp network) Console.WriteLine(); } +if (ShouldRun("pendulum")) +{ + Console.WriteLine("=== Pendulum-v1 — SAC from scratch (continuous control: torque ∈ [-2, 2]) ==="); + Console.WriteLine(" solved criterion (PRD): mean return >= -200 over 100 episodes"); + + var seeds = new SeedSequence(masterSeed); + var env = new PendulumEnv(); + var box = (BoxSpace)env.ActionSpace; + + ContinuousPolicyAgent agent; + if (TryLoadMlp("pendulum", "sac") is { } loaded) + { + agent = new ContinuousPolicyAgent(loaded, box.Dimensions, box.Low, box.High, seeds.CreateRng(RngStreams.Policy)); + } + else + { + var sw = System.Diagnostics.Stopwatch.StartNew(); + var result = SacTrainer.Train(env, new SacOptions + { + Hidden = [128, 128], + MaxSteps = 30_000, + WarmupSteps = 1_000, + EvalEvery = 5_000, + SolveThreshold = -150, + OnProgress = p => Console.WriteLine( + $" step {p.Step,6}/{p.MaxSteps} eval mean return: {p.EvalMeanReturn,8:F1} alpha: {p.Alpha:F3} qLoss: {p.CriticLoss:F2}"), + }, seeds); + sw.Stop(); + Console.WriteLine($"trained {result.StepsTrained:N0} env steps in {sw.Elapsed.TotalMinutes:F1} min"); + agent = result.Agent; + SaveMlp("pendulum", "sac", result.Actor); + } + + var eval = Evaluator.Evaluate(env, agent, episodes: 100, seeds.Derive(RngStreams.Evaluation)); + Console.WriteLine($"final greedy eval: {eval.MeanReturn:F1} mean return over 100 episodes " + + $"({(eval.MeanReturn >= -200 ? "SOLVED" : "not solved")})"); + Console.WriteLine(); + if (animate) AnimatePendulum(env, agent, seeds.Derive(RngStreams.Evaluation + 1)); + Console.WriteLine(); +} + Console.WriteLine("done."); return; +static void AnimatePendulum(PendulumEnv env, ContinuousPolicyAgent agent, ulong seed) +{ + Console.WriteLine("--- Pendulum — greedy playback (200 steps) ---"); + var (obs, _) = env.Reset(seed); + int frameTop = Console.CursorTop; + double episodeReturn = 0; + + for (int step = 0; step < PendulumEnv.DefaultMaxEpisodeSteps; step++) + { + Console.SetCursorPosition(0, frameTop); + Console.Write(env.RenderString()); + Console.WriteLine($" step {step,3}/200 return {episodeReturn,8:F1} "); + Thread.Sleep(30); + + var result = env.Step(agent.Act(obs, greedy: true)); + obs = result.Observation; + episodeReturn += result.Reward; + if (result.Done) break; + } + Console.WriteLine($" episode return: {episodeReturn:F1}"); +} + static (int SolvedInBudget, int SolvedAtAll) EvaluateRushHourGate(RushHourEnv env, GreedyQAgent agent, IReadOnlyList puzzles) { int solvedInBudget = 0, solvedAtAll = 0; diff --git a/src/RLDemo.Web/ClientApp/src/app/app.html b/src/RLDemo.Web/ClientApp/src/app/app.html index f248e28..82f2d5a 100644 --- a/src/RLDemo.Web/ClientApp/src/app/app.html +++ b/src/RLDemo.Web/ClientApp/src/app/app.html @@ -6,6 +6,7 @@ Rubik's Cube Snake Mountain Car + Pendulum Gallery diff --git a/src/RLDemo.Web/ClientApp/src/app/app.routes.ts b/src/RLDemo.Web/ClientApp/src/app/app.routes.ts index 9031ca3..4473226 100644 --- a/src/RLDemo.Web/ClientApp/src/app/app.routes.ts +++ b/src/RLDemo.Web/ClientApp/src/app/app.routes.ts @@ -7,6 +7,7 @@ export const routes: Routes = [ { path: 'cube', loadComponent: () => import('./cube/cube').then(m => m.Cube) }, { path: 'snake', loadComponent: () => import('./snake/snake').then(m => m.Snake) }, { path: 'mountaincar', loadComponent: () => import('./mountaincar/mountaincar').then(m => m.MountainCar) }, + { path: 'pendulum', loadComponent: () => import('./pendulum/pendulum').then(m => m.Pendulum) }, { path: 'gallery', loadComponent: () => import('./gallery/gallery').then(m => m.Gallery) }, { path: '**', redirectTo: '' }, ]; diff --git a/src/RLDemo.Web/ClientApp/src/app/home/home.ts b/src/RLDemo.Web/ClientApp/src/app/home/home.ts index 06feda7..07b5b13 100644 --- a/src/RLDemo.Web/ClientApp/src/app/home/home.ts +++ b/src/RLDemo.Web/ClientApp/src/app/home/home.ts @@ -58,6 +58,15 @@ import { RouterLink } from '@angular/router'; Play → + +

Pendulum

+

+ A SAC agent learned continuous control — a real-valued torque — to swing a rod upright and + balance it. Watch it live (the server drives the rod), or swing it yourself. +

+ Play → + +

Gallery

diff --git a/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum-api.ts b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum-api.ts new file mode 100644 index 0000000..e22a043 --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum-api.ts @@ -0,0 +1,37 @@ +import { Injectable } from '@angular/core'; + +/** One streamed frame of an AI Pendulum episode (PRD §7.1 principle B). `torque` is the continuous action applied. */ +export interface PendulumFrame { + cosTheta: number; + sinTheta: number; + angularVelocity: number; + torque: number; + reward: number; + done: boolean; +} + +export interface PendulumStatus { + status: 'loading' | 'training' | 'ready' | 'failed'; + trainingStep: number; + trainingMaxSteps: number; + lastEvalReturn: number; + error: string | null; +} + +@Injectable({ providedIn: 'root' }) +export class PendulumApi { + async status(): Promise { + const response = await fetch('/api/pendulum/status'); + return response.json(); + } + + /** Server-authoritative live stream — the backend drives the pendulum and pushes frames; the caller renders. */ + connectLive(onFrame: (frame: PendulumFrame) => void, onClose: () => void): WebSocket { + const proto = location.protocol === 'https:' ? 'wss' : 'ws'; + const socket = new WebSocket(`${proto}://${location.host}/api/pendulum/live`); + socket.onmessage = e => onFrame(JSON.parse(e.data) as PendulumFrame); + socket.onclose = onClose; + socket.onerror = onClose; + return socket; + } +} diff --git a/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum-logic.ts b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum-logic.ts new file mode 100644 index 0000000..f7a47a1 --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum-logic.ts @@ -0,0 +1,45 @@ +// Pure client-side Pendulum physics for HUMAN play (PRD §7.1: human mode is client-driven on a JS timer). +// Mirrors PendulumEnv's dynamics exactly so the human swings the same rod the AI does. + +const G = 10; +const M = 1; +const L = 1; +const DT = 0.05; +export const MAX_TORQUE = 2; +export const MAX_SPEED = 8; +export const MAX_STEPS = 200; + +const clamp = (x: number, lo: number, hi: number) => Math.min(hi, Math.max(lo, x)); + +export class PendulumGame { + theta = Math.PI; // start hanging down + thetaDot = 0; + steps = 0; + done = false; + + constructor() { this.reset(); } + + reset(): void { + this.theta = (Math.random() * 2 - 1) * Math.PI; // U[-π, π] + this.thetaDot = Math.random() * 2 - 1; // U[-1, 1] + this.steps = 0; + this.done = false; + } + + /** torque: continuous, clamped to ±MAX_TORQUE. Semi-implicit Euler, matching the env. */ + step(torque: number): void { + if (this.done) return; + const u = clamp(torque, -MAX_TORQUE, MAX_TORQUE); + let newThetaDot = this.thetaDot + (3 * G / (2 * L) * Math.sin(this.theta) + 3 / (M * L * L) * u) * DT; + newThetaDot = clamp(newThetaDot, -MAX_SPEED, MAX_SPEED); + this.theta += newThetaDot * DT; + this.thetaDot = newThetaDot; + this.steps++; + if (this.steps >= MAX_STEPS) this.done = true; + } + + /** Upright-ness in [0,1] for a simple human score (1 = perfectly balanced). */ + get upright(): number { + return (Math.cos(this.theta) + 1) / 2; + } +} diff --git a/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.html b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.html new file mode 100644 index 0000000..177034a --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.html @@ -0,0 +1,37 @@ +

Pendulum

+

+ Swing a frictionless rod upright and hold it there with a bounded torque. A SAC agent learned + this continuous-control task from scratch — the action is a real-valued torque, not a button. + Watch it (the server drives the rod live over a WebSocket) or swing it yourself + (runs in your browser). +

+ +@if (modelStatus(); as ms) { + @if (ms.status === 'training') { + + } @else if (ms.status === 'failed') { + + } +} + +
+ + +
+
+ + + +
+

{{ status() }}

+ @if (mode() !== 'idle') { +

Torque: {{ torque().toFixed(2) }}

+ } + @if (mode() === 'human') { +

Hold ← / → (or A / D) to apply torque. Pump back and forth to build the swing, then ease off near the top to balance.

+ } +
+
diff --git a/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.scss b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.scss new file mode 100644 index 0000000..24c2c6b --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.scss @@ -0,0 +1,39 @@ +:host { display: block; } + +.subtitle { max-width: 62ch; color: #9aa4b2; } + +.banner { + padding: 0.6rem 0.9rem; + border-radius: 6px; + margin: 0.5rem 0 1rem; + &.training { background: #2a2f3d; color: #ffd479; } + &.error { background: #3a2226; color: #ff8a8a; } +} + +.layout { + display: flex; + gap: 24px; + align-items: flex-start; + flex-wrap: wrap; +} + +.board { + background: #141823; + border-radius: 8px; + max-width: 100%; +} + +.side { + min-width: 220px; + + .actions { + display: flex; + flex-wrap: wrap; + gap: 8px; + margin-bottom: 0.5rem; + } + + .status { color: #cdd5e0; min-height: 1.4em; } + .torque { color: #ffd479; font-variant-numeric: tabular-nums; margin: 0.2rem 0; } + .hint { color: #8b95a5; font-size: 0.9em; } +} diff --git a/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.ts b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.ts new file mode 100644 index 0000000..2af8c77 --- /dev/null +++ b/src/RLDemo.Web/ClientApp/src/app/pendulum/pendulum.ts @@ -0,0 +1,152 @@ +import { Component, DestroyRef, ElementRef, afterNextRender, inject, signal, viewChild } from '@angular/core'; +import { PendulumApi, PendulumStatus } from './pendulum-api'; +import { MAX_TORQUE, PendulumGame } from './pendulum-logic'; + +/** + * Pendulum page (PRD §7.1): **Watch AI** = principle B (backend drives the episode over a WebSocket, this + * component renders frames), **Swing it yourself** = client-side physics on a JS timer with ←/→ applying a + * continuous torque. The SDK's continuous-control showcase — the action is a real-valued torque, not a button. + */ +@Component({ + selector: 'app-pendulum', + templateUrl: './pendulum.html', + styleUrl: './pendulum.scss', + host: { '(window:keydown)': 'onKeyDown($event)', '(window:keyup)': 'onKeyUp($event)' }, +}) +export class Pendulum { + private readonly api = inject(PendulumApi); + private readonly canvasRef = viewChild.required>('penCanvas'); + + protected readonly mode = signal<'idle' | 'watch' | 'human'>('idle'); + protected readonly status = signal('Watch the SAC agent swing the rod upright and balance it, or try it yourself.'); + protected readonly modelStatus = signal(null); + protected readonly torque = signal(0); + + private ctx: CanvasRenderingContext2D | null = null; + private socket: WebSocket | null = null; + private timer: ReturnType | null = null; + private game: PendulumGame | null = null; + private held = 0; // current human torque: -MAX_TORQUE, 0, or +MAX_TORQUE + + constructor() { + this.pollStatus(); + afterNextRender(() => { this.ctx = this.canvasRef().nativeElement.getContext('2d'); this.draw(-1, 0, 0); }); + inject(DestroyRef).onDestroy(() => this.stop()); + } + + protected async watchAi(): Promise { + this.stop(); + const st = await this.api.status().catch(() => null); + this.modelStatus.set(st); + if (!st || st.status !== 'ready') { + this.status.set(st?.status === 'training' ? 'The agent is still training — try again shortly.' : 'AI unavailable.'); + return; + } + this.mode.set('watch'); + this.status.set('Watching the SAC agent (the server drives the rod)…'); + this.socket = this.api.connectLive( + f => { + this.torque.set(f.torque); + this.draw(f.cosTheta, f.sinTheta, f.torque); + if (f.done) this.status.set('Episode complete — restarting…'); + }, + () => { if (this.mode() === 'watch') this.status.set('Stream closed.'); }); + } + + protected playHuman(): void { + this.stop(); + this.mode.set('human'); + this.held = 0; + this.status.set('Your turn — hold ← / → to apply torque. Pump it to swing up, then ease off to balance!'); + const g = this.game = new PendulumGame(); + this.draw(Math.cos(g.theta), Math.sin(g.theta), 0); + this.timer = setInterval(() => { + g.step(this.held); + this.torque.set(this.held); + this.draw(Math.cos(g.theta), Math.sin(g.theta), this.held); + if (g.done) { + this.status.set(`Time up — best balance held: ${(g.upright * 100).toFixed(0)}% upright.`); + this.clearTimer(); + } + }, 50); + } + + protected onKeyDown(event: KeyboardEvent): void { + if (this.mode() !== 'human') return; + if (event.key === 'ArrowLeft' || event.key === 'a') { this.held = -MAX_TORQUE; event.preventDefault(); } + else if (event.key === 'ArrowRight' || event.key === 'd') { this.held = MAX_TORQUE; event.preventDefault(); } + } + + protected onKeyUp(event: KeyboardEvent): void { + if (this.mode() !== 'human') return; + if (['ArrowLeft', 'ArrowRight', 'a', 'd'].includes(event.key)) this.held = 0; + } + + protected stop(): void { + this.socket?.close(); + this.socket = null; + this.clearTimer(); + this.game = null; + this.torque.set(0); + if (this.mode() !== 'idle') this.mode.set('idle'); + } + + private clearTimer(): void { + if (this.timer !== null) { clearInterval(this.timer); this.timer = null; } + } + + // cosTheta/sinTheta describe the rod angle (θ=0 is straight up); torque tints the applied-effort arc. + private draw(cosTheta: number, sinTheta: number, torque: number): void { + const ctx = this.ctx; + if (!ctx) return; + const cv = this.canvasRef().nativeElement; + const w = cv.width, h = cv.height; + const cx = w / 2, cy = h / 2; + const rodLen = Math.min(w, h) * 0.34; + + ctx.fillStyle = '#141823'; + ctx.fillRect(0, 0, w, h); + + // Reference circle the bob travels along. + ctx.strokeStyle = '#262c3a'; + ctx.lineWidth = 1; + ctx.beginPath(); ctx.arc(cx, cy, rodLen, 0, Math.PI * 2); ctx.stroke(); + + // Bob position: θ=0 points up. x = sinθ (right), y = -cosθ (up on screen). + const bx = cx + rodLen * sinTheta; + const by = cy - rodLen * cosTheta; + + // Rod — green when the bob is in the upper half (balanced), grey otherwise. + ctx.strokeStyle = cosTheta >= 0 ? '#7cffb2' : '#5b6678'; + ctx.lineWidth = 5; + ctx.beginPath(); ctx.moveTo(cx, cy); ctx.lineTo(bx, by); ctx.stroke(); + + // Torque arc at the pivot (direction + magnitude of effort). + if (Math.abs(torque) > 1e-3) { + ctx.strokeStyle = '#ffd479'; + ctx.lineWidth = 3; + const span = (Math.abs(torque) / MAX_TORQUE) * Math.PI; + ctx.beginPath(); + ctx.arc(cx, cy, 22, -Math.PI / 2, -Math.PI / 2 + (torque > 0 ? span : -span), torque < 0); + ctx.stroke(); + } + + // Pivot + bob. + ctx.fillStyle = '#9aa4b2'; + ctx.beginPath(); ctx.arc(cx, cy, 6, 0, Math.PI * 2); ctx.fill(); + ctx.fillStyle = cosTheta >= 0 ? '#7cffb2' : '#cdd5e0'; + ctx.beginPath(); ctx.arc(bx, by, 12, 0, Math.PI * 2); ctx.fill(); + } + + private pollStatus(): void { + void (async () => { + try { + const s = await this.api.status(); + this.modelStatus.set(s); + if (s.status === 'loading' || s.status === 'training') setTimeout(() => this.pollStatus(), 3000); + } catch { + // backend unreachable — leave status unknown + } + })(); + } +} diff --git a/src/RLDemo.Web/Controllers/MountainCarController.cs b/src/RLDemo.Web/Controllers/MountainCarController.cs index 0862e8f..f1c7465 100644 --- a/src/RLDemo.Web/Controllers/MountainCarController.cs +++ b/src/RLDemo.Web/Controllers/MountainCarController.cs @@ -51,6 +51,7 @@ await EpisodeStreamer.RunAsync( env, act: (obs, _) => agent.Act(obs, greedy: true), frame: (e, action, step) => new MountainCarFrameDto((float)e.Position, (float)e.Velocity, action, step.Reward, step.Done), + resetAction: -1, // -1 marks the freshly reset start state tickMs: 45, // smooth car motion startSeed: seed, ct: HttpContext.RequestAborted); diff --git a/src/RLDemo.Web/Controllers/PendulumController.cs b/src/RLDemo.Web/Controllers/PendulumController.cs new file mode 100644 index 0000000..b52dfe8 --- /dev/null +++ b/src/RLDemo.Web/Controllers/PendulumController.cs @@ -0,0 +1,62 @@ +using Microsoft.AspNetCore.Mvc; +using MintPlayer.AI.ReinforcementLearning.Environments; +using RLDemo.Web.Services; + +namespace RLDemo.Web.Controllers; + +/// One streamed frame of an AI Pendulum episode (PRD §7.1 principle B). Observation is the rod angle +/// as (cos θ, sin θ) plus angular velocity; Torque is the continuous action just applied. +public sealed record PendulumFrameDto(float CosTheta, float SinTheta, float AngularVelocity, float Torque, double Reward, bool Done); + +[ApiController] +[Route("api/pendulum")] +public sealed class PendulumController(PendulumModelService model) : ControllerBase +{ + private static int _seedCounter; + + [HttpGet("status")] + public StatusResponse Status() + { + _ = model.Agent; // touch: lazily loads a stored checkpoint + return new(model.Status.ToString().ToLowerInvariant(), + model.TrainingStep, model.TrainingMaxSteps, model.LastEvalReturn, model.Error); + } + + /// + /// Server-authoritative live stream of the trained AI balancing the pendulum (PRD §7.1 principle B): the + /// backend owns the episode + clock and pushes one frame per tick; the browser renders. 503 (rejected + /// upgrade) = still training. + /// Accepts GET (HTTP/1.1 Upgrade) AND CONNECT (HTTP/2 Extended CONNECT, RFC 8441) so the WebSocket + /// works over both — over HTTPS the browser negotiates HTTP/2, where a GET-only route returns 405. + /// + [AcceptVerbs("GET", "CONNECT", Route = "live")] + public async Task Live() + { + if (!HttpContext.WebSockets.IsWebSocketRequest) + { + Response.StatusCode = StatusCodes.Status400BadRequest; + return; + } + var agent = model.Agent; + if (agent is null) + { + Response.StatusCode = StatusCodes.Status503ServiceUnavailable; + return; + } + + using var socket = await HttpContext.WebSockets.AcceptWebSocketAsync(); + var env = new PendulumEnv(); + ulong seed = (ulong)System.Threading.Interlocked.Increment(ref _seedCounter); + await EpisodeStreamer.RunAsync( + socket, + env, + act: (obs, _) => agent.Act(obs, greedy: true), + frame: (e, action, step) => new PendulumFrameDto( + (float)Math.Cos(e.Theta), (float)Math.Sin(e.Theta), (float)e.AngularVelocity, + action.Length > 0 ? action[0] : 0f, step.Reward, step.Done), + resetAction: [0f], // zero torque marks the freshly reset start state + tickMs: 50, // smooth rod motion (matches the env's 0.05s dt) + startSeed: seed, + ct: HttpContext.RequestAborted); + } +} diff --git a/src/RLDemo.Web/Controllers/SnakeController.cs b/src/RLDemo.Web/Controllers/SnakeController.cs index b032381..53052cf 100644 --- a/src/RLDemo.Web/Controllers/SnakeController.cs +++ b/src/RLDemo.Web/Controllers/SnakeController.cs @@ -52,6 +52,7 @@ await EpisodeStreamer.RunAsync( act: (obs, mask) => agent.Act(obs, mask, greedy: true), frame: (e, action, step) => new SnakeFrameDto( [.. e.Body], e.Food, action, step.Reward, step.Done, e.FoodEaten, e.Length), + resetAction: -1, // -1 marks the freshly reset start state tickMs: 120, startSeed: seed, ct: HttpContext.RequestAborted); diff --git a/src/RLDemo.Web/Program.cs b/src/RLDemo.Web/Program.cs index 5ba31f5..c76719c 100644 --- a/src/RLDemo.Web/Program.cs +++ b/src/RLDemo.Web/Program.cs @@ -34,11 +34,13 @@ builder.Services.AddSingleton(); builder.Services.AddSingleton(); builder.Services.AddSingleton(); +builder.Services.AddSingleton(); builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddSingleton(sp => sp.GetRequiredService()); +builder.Services.AddSingleton(sp => sp.GetRequiredService()); builder.Services.AddSingleton(); // Integration tests control the model store themselves and host no SPA. diff --git a/src/RLDemo.Web/Services/EpisodeStreamer.cs b/src/RLDemo.Web/Services/EpisodeStreamer.cs index 71158ee..6be3ca6 100644 --- a/src/RLDemo.Web/Services/EpisodeStreamer.cs +++ b/src/RLDemo.Web/Services/EpisodeStreamer.cs @@ -19,15 +19,16 @@ namespace RLDemo.Web.Services; /// public static class EpisodeStreamer { - public static async Task RunAsync( + public static async Task RunAsync( WebSocket socket, TEnv env, - Func act, - Func, object> frame, + Func act, + Func, object> frame, + TAct resetAction, int tickMs, ulong startSeed, CancellationToken ct) - where TEnv : IEnvironment + where TEnv : IEnvironment { var masker = env as IActionMaskProvider; ulong seed = startSeed; @@ -36,13 +37,13 @@ public static async Task RunAsync( while (socket.State == WebSocketState.Open && !ct.IsCancellationRequested) { var (obs, _) = env.Reset(seed++); - // Frame with action = -1 marks the freshly reset start state so the client draws it. - await Send(socket, frame(env, -1, new StepResult(obs, 0, false, false, EnvInfo.Empty)), ct); + // The reset-marker action tags the freshly reset start state so the client draws it. + await Send(socket, frame(env, resetAction, new StepResult(obs, 0, false, false, EnvInfo.Empty)), ct); await Task.Delay(tickMs, ct); while (socket.State == WebSocketState.Open && !ct.IsCancellationRequested) { - int action = act(obs, masker?.CurrentActionMask()); + TAct action = act(obs, masker?.CurrentActionMask()); var step = env.Step(action); obs = step.Observation; await Send(socket, frame(env, action, step), ct); diff --git a/src/RLDemo.Web/Services/PendulumModelService.cs b/src/RLDemo.Web/Services/PendulumModelService.cs new file mode 100644 index 0000000..4c93087 --- /dev/null +++ b/src/RLDemo.Web/Services/PendulumModelService.cs @@ -0,0 +1,111 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Environments; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Training; +using MintPlayer.AI.ReinforcementLearning.Environments; + +namespace RLDemo.Web.Services; + +/// +/// Owns the Pendulum policy: loads it from the model store at startup, or trains one once with SAC (PLAN M23) +/// and saves it. Pendulum is the SDK's continuous-control showcase — a real-valued torque rather than a +/// discrete button — so it trains with SAC (off-policy, squashed-Gaussian actor + twin critics). Only the +/// actor is served (the deterministic tanh(mean) action); the critics and temperature are training-only. +/// +public sealed class PendulumModelService(IModelStore store, ILogger logger) : ITrainableModelService +{ + public const string EnvironmentId = "pendulum"; + public const string AlgorithmId = "sac"; + public const ulong TrainingMasterSeed = 1; + + public static SacOptions TrainingOptions(Action? onProgress = null) => new() + { + Hidden = [128, 128], + BufferCapacity = 200_000, + BatchSize = 256, + WarmupSteps = 1_000, + MaxSteps = 30_000, + EvalEvery = 5_000, + EvalEpisodes = 20, + SolveThreshold = -150.0, + OnProgress = onProgress, + }; + + private readonly object _lock = new(); + private ContinuousPolicyAgent? _agent; + + public ModelStatus Status { get; private set; } = ModelStatus.Loading; + public int TrainingStep { get; private set; } + public int TrainingMaxSteps { get; private set; } + public double LastEvalReturn { get; private set; } + public string? Error { get; private set; } + + /// The greedy policy agent, or null while not ready. Loads lazily from the store. + public ContinuousPolicyAgent? Agent + { + get + { + if (_agent is null && Status == ModelStatus.Loading) TryLoadFromStore(); + return _agent; + } + } + + private static ContinuousPolicyAgent WrapActor(Mlp actor) + { + var box = (BoxSpace)new PendulumEnv().ActionSpace; + return new ContinuousPolicyAgent(actor, box.Dimensions, box.Low, box.High, new Xoshiro256StarStar(TrainingMasterSeed)); + } + + public bool TryLoadFromStore() + { + lock (_lock) + { + if (_agent is not null) return true; + using var stream = store.TryOpenRead(EnvironmentId, AlgorithmId); + if (stream is null) return false; + _agent = WrapActor(MlpCheckpoint.Load(stream)); + Status = ModelStatus.Ready; + logger.LogInformation("Loaded Pendulum policy from the store."); + return true; + } + } + + public void EnsureModel(CancellationToken cancellationToken) + { + try + { + if (TryLoadFromStore()) return; + + logger.LogInformation("No Pendulum model in the store — training with SAC (a few minutes)..."); + lock (_lock) Status = ModelStatus.Training; + + var result = SacTrainer.Train(new PendulumEnv(), TrainingOptions(p => + { + cancellationToken.ThrowIfCancellationRequested(); + lock (_lock) + { + TrainingStep = p.Step; + TrainingMaxSteps = p.MaxSteps; + LastEvalReturn = p.EvalMeanReturn; + } + logger.LogInformation("Pendulum training: step {Step}/{Max}, eval {Eval:F1}", p.Step, p.MaxSteps, p.EvalMeanReturn); + }), new SeedSequence(TrainingMasterSeed)); + + store.Save(EnvironmentId, AlgorithmId, s => MlpCheckpoint.Save(result.Actor, s)); + _agent = WrapActor(result.Actor); + Status = ModelStatus.Ready; + logger.LogInformation("Pendulum model trained ({Steps} steps, eval {Eval:F1}) and saved.", result.StepsTrained, result.FinalEvalReturn); + } + catch (OperationCanceledException) + { + throw; + } + catch (Exception ex) + { + Status = ModelStatus.Failed; + Error = ex.Message; + logger.LogError(ex, "Pendulum model setup failed."); + } + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/PendulumApiTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/PendulumApiTests.cs new file mode 100644 index 0000000..69eed0a --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/PendulumApiTests.cs @@ -0,0 +1,70 @@ +using System.Net.Http.Json; +using System.Net.WebSockets; +using System.Text; +using System.Text.Json; +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using RLDemo.Web.Controllers; +using RLDemo.Web.Services; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +/// Pendulum WS-upgrade contract with no model: the live stream rejects the upgrade (503). +public class PendulumApiNoModelTests(PlaygroundFactory factory) : IClassFixture +{ + [Fact] + public async Task Live_WithoutModel_RejectsUpgrade() + { + var ws = factory.Server.CreateWebSocketClient(); + using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10)); + await Assert.ThrowsAnyAsync(() => + ws.ConnectAsync(new Uri(factory.Server.BaseAddress, "api/pendulum/live"), cts.Token)); + } +} + +/// Host fixture with a (deliberately untrained) Pendulum SAC actor in the store. +public class PendulumPlaygroundFactory : PlaygroundFactory +{ + public PendulumPlaygroundFactory() + { + // obs dim 3 → output 2 (mean + log-σ for the single torque dimension). + var actor = new Mlp([3, 32, 2], new Xoshiro256StarStar(7), Activation.Relu); + new FileModelStore(DataDirectory).Save( + PendulumModelService.EnvironmentId, PendulumModelService.AlgorithmId, s => MlpCheckpoint.Save(actor, s)); + } +} + +public class PendulumApiTests(PendulumPlaygroundFactory factory) : IClassFixture +{ + private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web); + + [Fact] + public async Task Status_ReportsReady_WhenSeeded() + { + var status = await factory.CreateClient().GetFromJsonAsync("/api/pendulum/status"); + Assert.NotNull(status); + Assert.Equal("ready", status.Status); + } + + [Fact] + public async Task Live_StreamsValidFrames() + { + var wsClient = factory.Server.CreateWebSocketClient(); + using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(20)); + using var socket = await wsClient.ConnectAsync(new Uri(factory.Server.BaseAddress, "api/pendulum/live"), cts.Token); + + var buffer = new byte[8 * 1024]; + for (int i = 0; i < 3; i++) + { + var result = await socket.ReceiveAsync(buffer, cts.Token); + var frame = JsonSerializer.Deserialize(Encoding.UTF8.GetString(buffer, 0, result.Count), Json); + Assert.NotNull(frame); + Assert.Equal(1.0, frame.CosTheta * frame.CosTheta + frame.SinTheta * frame.SinTheta, 3); // on the unit circle + Assert.InRange(frame.AngularVelocity, -8f, 8f); + Assert.InRange(frame.Torque, -2f, 2f); + } + + socket.Abort(); + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/PendulumEnvTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/PendulumEnvTests.cs new file mode 100644 index 0000000..6d0680f --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/PendulumEnvTests.cs @@ -0,0 +1,123 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Environments; +using MintPlayer.AI.ReinforcementLearning.Environments; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +public class PendulumEnvTests +{ + [Fact] + public void Reset_DrawsStartStateInBand_AndUnitCircleObs() + { + var env = new PendulumEnv(); + var (obs, _) = env.Reset(1); + + Assert.Equal(3, obs.Length); + Assert.Equal(1.0, obs[0] * obs[0] + obs[1] * obs[1], 4); // cos²+sin² == 1 + Assert.InRange(obs[2], -1f, 1f); // θ̇ drawn from [-1,1] + Assert.InRange(env.Theta, -Math.PI, Math.PI); + } + + [Fact] + public void ActionSpace_IsBoxTorque_PlusMinusTwo() + { + var box = Assert.IsType(new PendulumEnv().ActionSpace); + Assert.Equal(1, box.Dimensions); + Assert.Equal(-2f, box.Low[0]); + Assert.Equal(2f, box.High[0]); + } + + [Fact] + public void Step_AppliesGymnasiumDynamics() + { + var env = new PendulumEnv(); + env.Reset(1); + env.SetState(Math.PI / 2, 0.0); // rod horizontal, at rest + + var step = env.Step([0f]); // no torque + + // newθ̇ = θ̇ + (3g/2l·sinθ + 3/ml²·τ)·dt = 0 + (15·1 + 0)·0.05 = 0.75 + double expectedThetaDot = (3.0 * 10.0 / 2.0 * 1.0) * 0.05; + Assert.Equal(expectedThetaDot, env.AngularVelocity, 6); + Assert.Equal(Math.PI / 2 + expectedThetaDot * 0.05, env.Theta, 6); + Assert.Equal((float)expectedThetaDot, step.Observation[2], 5); + + // cost from the PRE-update angle (π/2): θ² + 0.1·θ̇² + 0.001·τ² = (π/2)² + Assert.Equal(-(Math.PI / 2) * (Math.PI / 2), step.Reward, 6); + } + + [Fact] + public void Step_ClampsTorqueToBounds_NoThrow() + { + var over = new PendulumEnv(); + over.Reset(1); over.SetState(0.0, 0.0); + var a = over.Step([5f]); // way past +2 + + var clamped = new PendulumEnv(); + clamped.Reset(1); clamped.SetState(0.0, 0.0); + var b = clamped.Step([2f]); + + Assert.Equal(b.Observation[2], a.Observation[2], 6); // identical: 5 was clamped to 2 + Assert.Equal(b.Reward, a.Reward, 6); + } + + [Fact] + public void Reward_IsNonPositive_AndZeroAtUprightRest() + { + var env = new PendulumEnv(); + env.Reset(1); env.SetState(0.0, 0.0); // upright, at rest + var step = env.Step([0f]); + Assert.Equal(0.0, step.Reward, 9); // the maximum attainable reward + } + + [Fact] + public void StepCap_TruncatesNeverTerminates() + { + var env = new PendulumEnv(); + env.Reset(1); + StepResult last = default; + for (int i = 0; i < PendulumEnv.DefaultMaxEpisodeSteps; i++) + last = env.Step([0f]); + + Assert.True(last.Truncated); + Assert.False(last.Terminated); // Pendulum has no terminal state + Assert.True(last.Done); + } + + [Fact] + public void SameSeed_ReproducesTrajectory() + { + var a = new PendulumEnv(); + var b = new PendulumEnv(); + var (oa, _) = a.Reset(42); + var (ob, _) = b.Reset(42); + Assert.Equal(oa, ob); + + for (int i = 0; i < 20; i++) + { + var sa = a.Step([0.5f]); + var sb = b.Step([0.5f]); + Assert.Equal(sa.Observation, sb.Observation); + Assert.Equal(sa.Reward, sb.Reward); + } + } + + [Fact] + public void SaveRestoreState_ResumesIdentically() + { + var a = new PendulumEnv(); + a.Reset(5); + for (int i = 0; i < 5; i++) a.Step([0.3f]); + var snapshot = a.SaveState(); + + var ra = a.Step([-0.7f]); + + var b = new PendulumEnv(); + b.Reset(999); + b.RestoreState(snapshot); + var rb = b.Step([-0.7f]); + + Assert.Equal(ra.Observation, rb.Observation); + Assert.Equal(ra.Reward, rb.Reward); + Assert.Equal(ra.Truncated, rb.Truncated); + } +} diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/SacTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/SacTests.cs new file mode 100644 index 0000000..84fe079 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/SacTests.cs @@ -0,0 +1,126 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Training; +using MintPlayer.AI.ReinforcementLearning.Environments; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +public class SacTests +{ + private static void AssertParamsEqual(IModule a, IModule b) + { + var pa = a.Parameters().ToArray(); + var pb = b.Parameters().ToArray(); + Assert.Equal(pa.Length, pb.Length); + for (int i = 0; i < pa.Length; i++) Assert.Equal(pa[i].Data, pb[i].Data); + } + + [Fact] + public void ConcatCols_RoutesGradientToEachHalf() + { + var a = new Tensor([1f, 2f, 3f, 4f], 2, 2) { RequiresGrad = true }; + var b = new Tensor([5f, 6f], 2, 1) { RequiresGrad = true }; + + var cat = a.ConcatCols(b); + Assert.Equal(3, cat.Cols); + Assert.Equal([1f, 2f, 5f, 3f, 4f, 6f], cat.Data); // rows interleaved correctly + + cat.Sum().Backward(); + Assert.Equal([1f, 1f, 1f, 1f], a.Grad!); + Assert.Equal([1f, 1f], b.Grad!); + } + + [Fact] + public void SliceCols_ScattersGradientBackToSourceColumns() + { + var x = new Tensor([1f, 2f, 3f, 4f, 5f, 6f, 7f, 8f], 2, 4) { RequiresGrad = true }; + var mid = x.SliceCols(1, 2); + Assert.Equal([2f, 3f, 6f, 7f], mid.Data); + + mid.Sum().Backward(); + Assert.Equal([0f, 1f, 1f, 0f, 0f, 1f, 1f, 0f], x.Grad!); // only the sliced columns receive grad + } + + [Fact] + public void Normal_RSample_IsBoundedAndLogProbFinite() + { + var mean = new Tensor([0.0f, 0.0f], 1, 2); + var logStd = new Tensor([-0.5f, -0.5f], 1, 2); + var dist = new Normal(mean, logStd); + + var (action, logProb) = dist.RSample(new Xoshiro256StarStar(3)); + Assert.Equal([1, 2], action.Shape); + Assert.All(action.Data, v => Assert.InRange(v, -1f, 1f)); // tanh-squashed + Assert.Single(logProb.Data); + Assert.True(float.IsFinite(logProb.Data[0])); + + Assert.Equal(MathF.Tanh(0f), dist.Mode().Data[0], 6); // greedy = tanh(mean) + } + + [Fact] + public void Normal_FromNetOutput_SplitsAndClampsLogStd() + { + // Output [mean(2) | logStd(2)] with a huge log-σ that must clamp to LogStdMax before exp. + var netOut = new Tensor([0.1f, -0.2f, 50f, 50f], 1, 4); + var dist = Normal.FromNetOutput(netOut, 2); + var (_, logProb) = dist.RSample(new Xoshiro256StarStar(1)); + Assert.True(float.IsFinite(logProb.Data[0])); // would be NaN/Inf if log-σ weren't clamped + } + + private static SacOptions ResumeOptions(int steps) => new() + { + Hidden = [32, 32], + BufferCapacity = 5_000, + WarmupSteps = 200, + BatchSize = 32, + EvalEvery = int.MaxValue, // no eval interruptions → clean determinism + MaxSteps = steps, + }; + + [Fact] + public void Sac_ResumesBitwiseIdentically() + { + const ulong masterSeed = 123; + var resultA = SacTrainer.Train(new PendulumEnv(), ResumeOptions(3_000), new SeedSequence(masterSeed)); + + var resultB1 = SacTrainer.Train(new PendulumEnv(), ResumeOptions(1_500), new SeedSequence(masterSeed)); + using var stream = new MemoryStream(); + resultB1.State.Save(stream); + stream.Position = 0; + var restored = SacTrainingState.Load(stream); + var resultB2 = SacTrainer.Train(new PendulumEnv(), ResumeOptions(3_000), new SeedSequence(masterSeed), resume: restored); + + AssertParamsEqual(resultA.Actor, resultB2.Actor); + AssertParamsEqual(resultA.Critic1, resultB2.Critic1); + AssertParamsEqual(resultA.State.Target1, resultB2.State.Target1); + } + + [Fact] + [Trait("Category", "Slow")] + public void Sac_SolvesPendulum_MedianOf3Seeds() + { + // RL is statistical — single-seed pass/fail lies (PRD §3), so gate on the median of 3. A competent + // SAC reaches a mean return around −150 on Pendulum; random ≈ −1200. Gate generously at −250. + var finals = new List(); + foreach (ulong seed in new ulong[] { 1, 2, 3 }) + { + var seeds = new SeedSequence(seed); + var result = SacTrainer.Train(new PendulumEnv(), new SacOptions + { + Hidden = [128, 128], + MaxSteps = 25_000, + WarmupSteps = 1_000, + EvalEvery = 5_000, + SolveThreshold = -150, + }, seeds); + var eval = Evaluator.Evaluate(new PendulumEnv(), result.Agent, episodes: 50, seeds.Derive(RngStreams.Evaluation)); + finals.Add(eval.MeanReturn); + } + + finals.Sort(); + Assert.True(finals[1] >= -250, + $"median SAC eval {finals[1]:F1} (all: {string.Join(", ", finals.Select(f => f.ToString("F0")))})"); + } +}