M23: Pendulum + SAC — continuous control (first Box action space)

[PieterjanDeClippel]

What

Adds the SDK's continuous-action pillar — every prior agent picks one of N discrete actions; SAC outputs a real-valued torque. Showcased by Pendulum-v1.

Investigated by a 4-agent team (autograd seam, Box/env seam, SAC design, web/gates/docs), then built + measured the same day.

Core (no new backend kernels needed)

• Tensor.ConcatCols / SliceCols — two small column-wise autograd ops (pure memory layout). Concat feeds the critic its concat(state, action) input (routing the actor's gradient through the action half); Slice splits the policy net's [B,2A] output into the mean / log-σ halves.
• Nn/Normal — diagonal squashed Gaussian (the continuous mirror of Categorical): reparameterized tanh(mean+σ·ε) sample, log-prob with the tanh change-of-variables correction, Mode=tanh(mean). ε is a detached RandomNormal leaf (stop-gradient for free); log-σ clamped to [-20,2].
• ContinuousReplayBuffer (+ checkpoint) — float-vector actions, no mask; the discrete ReplayBuffer stays byte-compatible.
• SacTrainer — squashed-Gaussian actor, twin Q-critics (clipped double-Q), Polyak soft target update (a SoftUpdate helper, not a CopyFrom overload), auto-tuned entropy temperature. ContinuousPolicyAgent serves greedy = tanh(mean) rescaled to the box bounds. Bitwise-resumable SacTrainingState. The off-policy loop mirrors DqnTrainer.

Env

PendulumEnv (IEnvironment<float[],float[]> + IStatefulEnvironment) — faithful Gymnasium Pendulum-v1: obs [cos θ, sin θ, θ̇], action torque ∈ [−2,2] (Box(1)), semi-implicit Euler, pre-update cost, torque clamp, no terminal (truncate@200).

Web (PRD §7.1 principle B)

• EpisodeStreamer generalized over TAct (MountainCar/Snake pass resetAction: -1).
• PendulumModelService (SAC), PendulumController (WS /api/pendulum/live, h2 Extended CONNECT).
• Angular page: <canvas> rod + Watch AI (WebSocket) / Swing it yourself (client TS physics on a JS timer, ←/→ = continuous torque). Nav + home card + route.
• Seed models/pendulum.sac.ckpt (Git LFS), served actor-only via the existing MlpCheckpoint.

Measured

• SAC (Hidden [128,128], 30k steps, ~6 min CPU) → greedy eval −149.1 over 100 episodes (random ≈ −1200; past the −200 solved bar and the −150 solve-threshold).
• Bitwise-resume test green; 270 fast tests pass; both web modes verified live in-browser (Playwright, no console errors).
• Slow median-of-3-seeds gate (Sac_SolvesPendulum, gate ≥ −250) included, opt-in.

Docs

PRD (algorithm ladder, non-goals clarification, env table, §7.1 interaction table), PLAN (M23 milestone, breadth list, gate-results table, console help), ADDING_A_GAME (continuous-control seam note).

🤖 Generated with Claude Code