diff --git a/README.md b/README.md index b6f5471..088d36e 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ See also: | `src/RLDemo.Console` | Console demo — watch agents learn and play (`--save`/`--load` persist trained models) | | `src/RLDemo.Web` | **MintPlayer.AI.ReinforcementLearning Playground** — ASP.NET Core + Angular web app: three games (Rush Hour, classic-feel 2048, a 3D Rubik's Cube), each playable yourself and solvable by the trained AI with step-through playback, plus a public gallery of every submitted board | | `tests/MintPlayer.AI.ReinforcementLearning.Tests` | xUnit suite incl. solved-threshold gates, determinism tests and web API integration tests | -| `tools/MintPlayer.AI.ReinforcementLearning.Lab` | Long-running imitation-learning campaigns (Rush Hour from the BFS oracle, Rubik's Cube from Kociemba) — resumable, checkpointing into the model store | +| `tools/MintPlayer.AI.ReinforcementLearning.Lab` | Long-running training campaigns (Rush Hour, Rubik's Cube, Snake, FruitCake) — resumable, checkpointing into the model store. Add `--viz` for a live in-browser **network visualizer** (see below) | ## Run the playground @@ -120,6 +120,36 @@ checkpoints live (it re-reads them every few minutes), or at `models/` to refres committed seeds. The DQN fallbacks can be retrained from the console: `dotnet run --project src/RLDemo.Console -c Release -- rushhour|cube --save --data models`. +### Training flags + +Every campaign is launched by `tools/MintPlayer.AI.ReinforcementLearning.Lab` and shares a common set of flags, +plus a few per-game knobs. + +**Common (all `--game`s):** + +| Flag | Meaning | +|---|---| +| `--game ` | `rushhour` (default), `snake`, `fruitcake`, `cube`, `cube-policy`, `cube-davi` | +| `--hours H` | wall-clock training budget | +| `--data DIR` | model-store directory to read/write checkpoints (e.g. `src/RLDemo.Web/data` or `models`) | +| `--seed S` | RNG seed (runs are reproducible) | +| `--lr LR` | learning rate | +| `--eval-only` | evaluate the stored net and exit — no training | +| `--viz [port]` | live [network visualizer](#watch-the-network-train-live-visualizer) (Development only; bare `--viz` = 5250) | +| `--grow` / `--grow-every N` | progressively grow the net wider+deeper (all games **except** `cube-davi`, which grows via `--auto-widen`) | + +**Per-game knobs:** + +| Game | Flags | +|---|---| +| `snake`, `fruitcake` (DQN) | `--steps N` (absolute step cap), `--chunk-steps N`, `--episodes N` (eval games), `--explore E` (ε-start), `--hidden a,b` (trunk widths), `--gamma G` | +| `snake` (extra) | `--train-grid`, `--eval-grid`, `--step-penalty`, `--safe-mask`; `--search` evaluates the look-ahead planner (with `--depth`/`--beam`/`--w-*`/`--net`) instead of training | +| `fruitcake` (extra) | `--nstep N`, `--shape` (reward shaping), `--noisy` (NoisyNets); offline comparison modes `--ab` (`--baseline`, `--ab-episodes`) and `--search-eval` (`--depth`/`--topk`/`--topk2`/`--leaf`) | +| `cube` (imitation) | `--width W` (trunk width) | +| `cube-policy` (EfficientCube) | `--width W`, `--max-scramble D`, `--beam W`, `--episodes N` | +| `rushhour` (imitation) | common flags only | +| `cube-davi` (self-taught value net) | its own set — see the section below (`--net residual`, `--width`, `--max-depth`, `--samples`, `--auto-widen`, `--grow-to`, and the `--eval-only --search --batched …` measurement mode) | + ### Train / further-train the self-taught cube solver (DAVI) The cube's strongest net is trained *teacher-free* by deep approximate value iteration @@ -153,6 +183,50 @@ dotnet run --project tools/MintPlayer.AI.ReinforcementLearning.Lab -c Release -- Full recipe + expected single-GPU wall-clock: [`tools/…/Lab/CUBE_CAMPAIGN.md`](tools/MintPlayer.AI.ReinforcementLearning.Lab/CUBE_CAMPAIGN.md). +## Watch the network train (live visualizer) + +Add `--viz` to **any** Lab training run to open a live, in-browser view of the neural network as it learns: + +``` +dotnet run --project tools/MintPlayer.AI.ReinforcementLearning.Lab -c Release -- --game snake --viz +``` + +Open the printed URL (default `http://localhost:5250`). The page draws the network as a node-link graph +with per-layer weight heatmaps and a loss sparkline, all repainting live — you watch the weights move +from random init toward a policy. **Hover any neuron, connection, or heatmap** for a plain-language +explanation: each input names the observation feature it is (and its current value), each output the +action it controls (and its live Q-value/score), and hidden neurons show their current activation. Works +for every game — `snake`, `fruitcake`, `rushhour`, `cube`, `cube-policy`, `cube-davi`. + +![Live network visualizer](docs/screenshots/m36-network-visualizer.png) + +**Watch the net grow, too.** The DQN games can grow their network *wider and deeper* mid-training +(function-preserving Net2WiderNet / Net2DeeperNet). Add `--grow` and watch the graph add units and layers live: + +``` +dotnet run --project tools/MintPlayer.AI.ReinforcementLearning.Lab -c Release -- --game fruitcake --viz --grow +``` + +It starts from a tiny net and grows through `[16] → [32] → [32,32] → … → [128,128,128]` with no loss spike (each +step preserves the function exactly). `--grow-every N` sets how many training steps between growth steps — slow it +down (and open the page immediately) to catch every stage from the start: + +``` +dotnet run --project tools/MintPlayer.AI.ReinforcementLearning.Lab -c Release -- \ + --game fruitcake --viz --grow --grow-every 5000 +``` + +`--grow` works for **every trainable game** — the DQN nets (Snake, FruitCake) and the two-headed imitation policy +nets (`--game cube|rushhour --grow`); the cube's self-taught DAVI value net already grows its width during its +campaign (`--auto-widen`). + +![The network after growing wider and deeper](docs/screenshots/m36-network-grows.png) + +It is a **development-only** tool: the socket only starts in a Development host environment (the Lab +defaults to it; set `DOTNET_ENVIRONMENT=Production` to disable) and is never part of the deployed web app. +Telemetry is read-only — a watched run trains bitwise-identically to an unwatched one. Design notes: +[docs/prd/NETWORK_VISUALIZER_PRD.md](docs/prd/NETWORK_VISUALIZER_PRD.md). + ## Run the tests ``` diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fd95056..7a5c00f 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -333,6 +333,8 @@ stops early) and **score-maximizing** (eval = mean return; runs to the wall-cloc | [`tools/…Lab/FruitCakeAb.cs`](../tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeAb.cs) | Paired-seed A/B of two nets → mean±SD, paired Δ±SE, verdict. | | [`tools/…Lab/FruitCakeSearchEval.cs`](../tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeSearchEval.cs) | Same net, **search vs greedy** on paired seeds → score + max-tier distribution + watermelon count. | | [`tools/…Lab/CampaignCli.cs`](../tools/MintPlayer.AI.ReinforcementLearning.Lab/CampaignCli.cs) | `ConsoleAndCsv(path)` — the `OnEval` IO bridge (console + CSV). | +| [`Core/Telemetry/NetworkTelemetry.cs`](../src/MintPlayer.AI.ReinforcementLearning.Core/Telemetry/NetworkTelemetry.cs) | Live-network telemetry seam (**pull** model): `INetworkTelemetrySource` (`NetKind`/`SnapshotParameters()`/`Sample()`) + `NetworkInspector` (describes/snapshots any net from its parameter tensors; bounded ≤24² weight heatmap). Read-only — training stays bitwise-identical. Every campaign implements the source. | +| [`tools/…Lab/VizServer.cs`](../tools/MintPlayer.AI.ReinforcementLearning.Lab/VizServer.cs) + [`VizLauncher.cs`](../tools/MintPlayer.AI.ReinforcementLearning.Lab/VizLauncher.cs) | `--viz [port]` live viewer: an `HttpListener` on localhost serving one self-contained Canvas 2D page (node-link graph + heatmaps + **beginner hover tooltips**) + a **WebSocket** (`/ws`). Fully async (per-viewer `Channel` + async pump + async sample loop). `VizLauncher` is the shared flag handler, **gated to a Development environment**. Bidirectional-ready for future viewer→trainer controls. | ```bash # Train (score-maximizing, resumable): writes /.dqn.ckpt (deployable, save-best) + .dqn-state.ckpt (resume) @@ -349,6 +351,30 @@ dotnet run -c Release --project tools/MintPlayer.AI.ReinforcementLearning.Lab -- --game fruitcake --search-eval --data ./data --depth 3 --topk 5 --topk2 2 ``` +**Grow the net mid-training (M37).** Function-preserving growth (Net2WiderNet/Net2DeeperNet) is factored into +`Core/Nn/Net2Net.cs` (`WidenTrunk` splits duplicated units' outgoing weights; `SetIdentity` inserts an identity layer) +and shared by every trainable net. `--grow` (`[16]→…→[128,128,128]` schedule, watch it in `--viz`) works for: **DQN** +games via `DuelingQNet.WidenTo`/`.Deepen` + `DqnGrowth` (rebuilds state via `DqnTrainingState.WithNetwork`); the +**imitation/EfficientCube policy nets** — `CubePolicyNet`/`RushHourPolicyNet` were refactored onto a shared +variable-depth `PolicyValueNet` core — via `PolicyGrowth` (their checkpoint is now v2 with a trunk-widths array; v1 +shipped files still load). DAVI's `ResidualMlp` already grows width (`--auto-widen`/`--grow-to`). *Change it:* schedule +in `DqnGrowth.Stages`; operators in `Net2Net`. + +**Watch a net train (M36).** Add `--viz [port]` to **any** game (`--game snake --viz`, bare = 5250; works for all six +`--game`s): the Lab hosts a localhost page that shows the net's topology + a live weight snapshot over a **WebSocket** +(`/ws`), so you watch the weights move from random init toward a policy in-browser — with hover tooltips explaining +each part. It's a **pull** model: each campaign implements `INetworkTelemetrySource`, and `VizServer`'s async sample +loop reads `SnapshotParameters()`/`Sample()` on a cadence. A source may also expose `InputLabels`/`OutputLabels` + +`SampleIo()` + `SampleActivations()` (default null) to name each neuron and stream its live value. Labels live on the +envs (`SnakeEnv`, `FruitCakeEnv`, `RubiksCubeEnv`, `RushHourBoard`); every neuron — input, hidden (each net exposes +`LayerActivations`), and output — shows a live value. The DQN games forward their actual `CurrentObs`; the +batch-trained nets (Cube/Rush Hour/DAVI) forward a fixed probe state (see `CubeViz`). The viewer attaches output +labels to the column matching their count (an action head can precede a value head) and draws labeled columns in full. Reading the parameter `float[]`s never mutates anything, so +a viz run is byte-identical to a no-viz one (SHA256-verified). The socket is **gated to a Development environment** +(`VizLauncher`); dev-only, never in the deployed web app. See `prd/NETWORK_VISUALIZER_PRD.md`. *Change it:* frame +contents/heatmap in `NetworkInspector`; each campaign's `SnapshotParameters()`/`Sample()`; page/tooltips/transport in +`VizServer`; the `--viz` gate in `VizLauncher`. + **Resume contract:** a campaign saves a *deployable* net (`.dqn.ckpt`, **save-best guarded** for noisy DQN eval) and a *full resume state* (`.dqn-state.ckpt`: optimizer + replay buffer + RNG + env snapshot). Re-running continues bitwise-identically. The cube's self-taught DAVI campaign has its own recipe in diff --git a/docs/prd/NETWORK_VISUALIZER_PRD.md b/docs/prd/NETWORK_VISUALIZER_PRD.md new file mode 100644 index 0000000..ff078a3 --- /dev/null +++ b/docs/prd/NETWORK_VISUALIZER_PRD.md @@ -0,0 +1,209 @@ +# Network Visualizer — see the net, and watch it learn — PRD + +**Status:** M36.1 complete (+ M37 net growth) — PR #30, ready to merge · 2026-07-12 · branch `m36-network-visualizer` (off `master`) +**Owner:** Pieterjan +**Milestone:** [PLAN.md](PLAN.md) M36 · **Depends on:** the Core NN + checkpoint layer (§2/§5 of [../ARCHITECTURE.md](../ARCHITECTURE.md)) and the Lab training harness (§8) — additive, no change to existing training behaviour. + +## 1. Problem + +The project trains a family of neural networks (`Mlp`, `DuelingQNet`, `ResidualMlp`, the two-headed policy nets) and +ships them as versioned binary `.ckpt` files, but a network is only ever visible as **numbers**: a checkpoint on +disk, a CSV of eval metrics, a line of console output per evaluation. There is no way to *see* a network — its +shape, the structure in its weights — and, more importantly, **no way to watch a network change as it is being +trained.** You start a campaign, wait minutes per eval, and read a scalar. The learning itself is invisible. + +We want two things: + +1. **Inspect a `.ckpt`** — load any shipped checkpoint and see its architecture (layers, widths) and its weights. +2. **Watch a network evolve while it trains** *(the priority)* — a live view, updating continuously as the training + loop runs, so you can literally see the weights move from random init toward a learned policy. + +## 2. Goal & success criteria + +- **Live gate (the priority, human judgement, in-browser):** start any game's training run with `--viz`; open the + served page; **the network's weights visibly change as training proceeds** — edges and per-layer heatmaps repaint + on a steady cadence, and the metric read-outs (step, loss, eval, ε) advance. Opening the page *mid-run* immediately + shows the current graph, then continues live. Closing/refreshing the browser never disturbs training. +- **All games:** `--viz` works for **every trainable game** (`snake`, `fruitcake`, `rushhour`, `cube`, `cube-policy`, + `cube-davi`) — DQN, imitation-policy, EfficientCube-policy, and DAVI value nets alike — through one generic seam, + with no per-trainer code. +- **Beginner-friendly & environment-aware:** hovering any neuron, connection, or heatmap shows a plain-language + tooltip. Each **input** names the exact observation feature it is (FruitCake: "Column 7: surface height"; Snake: + "Vision: is a wall/body 4 up, 1 left?"), each **output** names the action it controls (Snake: "Move Up"; Cube: + "R' — turn the Right face counter-clockwise"; Rush Hour: "Move the red car one cell up/left"; DAVI: "Estimated + distance to solved"), and **every neuron shows a live value** — input feature values, output Q-values/scores, and + **hidden-neuron activations** — for all five trainable games. The DQN games (Snake, FruitCake) forward their actual + current observation; the batch-trained nets (Cube, Rush Hour, DAVI), which have no running env, forward a **fixed + probe state** each frame, so you watch the net's opinion of one specific board evolve as it learns. +- **Dev-only, gated:** the live socket only comes up in a **Development** host environment; a Production-configured + process ignores `--viz` (with a note). It is never part of the deployed web app. +- **Zero training impact:** attaching the visualizer leaves training **bitwise-identical** (verified: viz vs no-viz + checkpoints are SHA256-equal) — sampling is a read-only observation of the parameter tensors on a background + thread; it never touches the RNG, optimizer, or net, and costs nothing while no browser is connected. +- **Fully async & dependency-light:** no blocking network I/O on any path (per-viewer async send queue + async + sample loop), and no new NuGet/npm packages (uses `HttpListener`'s built-in WebSocket support + the browser's + native `WebSocket`); the viewer page is a single self-contained HTML file (inline CSS/JS, no external requests). + +**Non-goals (v1).** No editing/altering weights from the UI. No viewer→trainer control messages *yet* (the WebSocket +is chosen so they can be added without changing transport). No gradient-flow animation. Hidden-neuron +activations and live values are shown for every trainable game — the batch-trained nets (Cube, Rush Hour, DAVI) via a +fixed probe state rather than their actual training batch (see §7). No 3-D layout. No serving the live stream from the public +web app (training is dev-side; see §3). No diffing two checkpoints. Continuous-control PPO/SAC (Pendulum, MountainCar) +aren't trained through the Lab's `--game` dispatch, so they're out of scope until they are. + +## 3. Key decision — stream from the *training process*, not the web app + +An investigation (4 parallel agents, 2026-07-12) established the load-bearing constraints: + +- **The web app never trains and has no server WebSocket infra.** `RLDemo.Web` is load-only; the + server-authoritative "Watch AI" WebSocket stack was deliberately **removed in M32/M33** — every game now computes + client-side over REST. Reintroducing server sockets into the web app to watch *training* would cut against that + grain and, worse, the web app isn't where training happens. +- **Training lives only in the Lab CLI** (`tools/…Lab`, `CampaignRunner` on `AIHost`). That process owns the net, + the optimizer, and the loop — it is the only place the weights actually move. +- **Reading weights mid-training is free on the CPU path.** Every game net's parameters are host-resident + `float[]` (`Tensor.Data`); reading them perturbs nothing. (The GPU-resident DAVI cube path keeps a CPU master net + that's synced periodically — the viewer reads that, so it shows the last-synced weights with no device lock.) + +So the live view is **served by the training process itself**: the Lab hosts a tiny `HttpListener` on localhost +that serves one HTML page (`GET /`) and streams telemetry to it over a **WebSocket** (`GET /ws`). The static-inspection +mode has a different, equally natural home: the browser (**Angular `/network` route**) already ships `.ckpt` parsers +(`snake-net.ts` etc.), so it can inspect a checkpoint entirely client-side with no server at all. + +**Transport — WebSocket (owner's call, 2026-07-12).** Telemetry today is one-way (server→client), which Server-Sent +Events models with the least code — no handshake, browser auto-reconnect for free. We chose a **WebSocket** anyway, +deliberately, because the channel is meant to become **bidirectional**: the viewer should be able to send control +messages back to the training run (pause/step, change the frame cadence, select a layer, request an on-demand +snapshot) without swapping transports later. `HttpListener` supports the server side natively +(`AcceptWebSocketAsync`) and the browser has `WebSocket` built in, so there's no new dependency; the only cost over +SSE is the framing/lifecycle we manage in `VizServer` (send serialized per socket + bounded by a timeout, a receive +loop that also drains future control messages, and a small client-side reconnect). The removed-in-M32/M33 server WS +stack was *web-app* infra; this is a dev-only server in the training process and doesn't reintroduce it there. + +**Design-it-twice note.** The other alternative — persist per-step snapshots to a JSONL/`.ckpt` sidecar and have a +separate viewer tail the file — was rejected for the live path: it doubles the I/O, adds a polling viewer, and still +needs a transport for "push me the latest." Streaming from the training process is strictly simpler and genuinely +live. The sidecar idea survives only as the trivial static case (read one `.ckpt`). + +## 4. Design — a read-only *pull* seam + a viewer served by the trainer + +### 4.A Core: the telemetry source (`Core/Telemetry/NetworkTelemetry.cs`) + +A **pull** model, not a push: the campaign publishes what it has; a viewer samples it on its own cadence. This is +what makes "all games" free — no trainer calls anything. + +```csharp +public interface INetworkTelemetrySource +{ + string NetKind { get; } // e.g. "dueling-q", "cube-policy", "value-davi-res" + IReadOnlyList? SnapshotParameters(); // current weights+biases, or null until the net exists + NetworkMetrics Sample(); // step / maxSteps / loss / eval / epsilon (NaN where N/A) + // Optional semantics (default null → generic tooltips): + IReadOnlyList? InputLabels => null; // name of each input feature + IReadOnlyList? OutputLabels => null; // name of each action/output + (float[] Input, float[] Output)? SampleIo() => null; // current observation + the net's output for it + float[][]? SampleActivations() => null; // each layer's live activation vector +} +``` + +The label/IO members are **default-implemented** (null), so a game opts in without every other campaign changing. +Labels are published on the environments (`FruitCakeEnv.ObservationLabels`/`ActionLabels`, `SnakeEnv`, +`RubiksCubeEnv.ActionLabels` = the 12 quarter-turns, `RushHourBoard.ActionLabels` = the 32 vehicle×dir moves) and +attached by whichever campaign owns that net. The campaign's `SampleIo()`/`SampleActivations()` +forward one observation through the net to get per-action Q-values/scores and each hidden layer's activations +(`DuelingQNet`/`CubePolicyNet`/`RushHourPolicyNet`/`ResidualMlp` each expose a `LayerActivations`). The DQN games +forward their actual `DqnTrainingState.CurrentObs`; the batch-trained nets (Cube, Rush Hour, DAVI) have no running env, +so they forward a **fixed probe state** (a constant-seed depth-8 scramble / the level-1 puzzle) built once. Every +forward is read-only (no `Backward`) so it can't perturb training — verified SHA256-identical **with a viewer +connected** — and for a single row it stays on the CPU even under the GPU backend, so it never contends with a +GPU training step. + +- `NetworkInspector.Describe(parameters, kind)` / `.CaptureFrame(parameters, metrics)` turn **any** net's parameter + tensors into telemetry by pairing each rank-2 weight with the rank-1 bias that follows it — the layout every net in + the library emits — so no per-architecture code is needed. Keying off `Parameters()` (not a concrete type) is why + the non-`IModule` policy nets (`CubePolicyNet`, `RushHourPolicyNet`) work too. +- A frame carries, per layer: weight min/max/mean-|w|/L2, bias mean-|w|, and a **downsampled magnitude heatmap** + (block-averaged |w| capped at 24×24). The cap makes a frame's size independent of the true matrix size — a + 1024-wide layer streams the same few KB as a 16-wide one. The source, not the viewer, bounds the payload. + +### 4.B Campaign wiring — one interface, six games + +Each `ITrainingCampaign` also implements `INetworkTelemetrySource` in ~4 lines: return the live net's parameters +(`_state.Online` for DQN, `_net` for the policy/DAVI campaigns) and a `NetworkMetrics` from fields it already tracks +(step, last loss, eval). No trainer changes at all. Sampling reads the parameter arrays on the viewer's background +thread — a benign race with the training thread's writes (a torn float only flickers one heatmap pixel; never a +fault), and provably harmless to training (no writes, no RNG, no ordering) — hence bitwise-identical checkpoints. + +### 4.C Live viewer — the Lab's `--viz` server (`tools/…Lab/VizServer.cs` + `VizLauncher.cs`) + +`VizServer` starts an `HttpListener` on `http://localhost:/` and runs a background **sample loop** that pulls +the source every ~150 ms and broadcasts: + +- `GET /` → one self-contained HTML page (Canvas 2D): a node-link graph (layers as columns of neurons, capped for + display; edge brightness = weight magnitude) + a per-layer weight-heatmap strip + a loss sparkline, and — for + newcomers to neural nets — a **hover tooltip** on every neuron / connection / heatmap. Labeled input/output + columns are drawn in full (not capped) so each neuron is individually hoverable and shows its name + live value; + the canvas grows in height to fit. Unlabeled/hidden columns stay capped with generic wording. +- `GET /ws` → the WebSocket. Messages self-describe (`{"type":"topology"|"frame","data":…}`). Topology is broadcast + only when the shape changes (e.g. a DAVI widen/grow); the latest is retained so a mid-run joiner gets the graph at + once. The client auto-reconnects on drop. +- **Fully async, fault-tolerant:** each viewer has a bounded outbound `Channel` drained by an async send pump (no + blocking sends; a slow viewer just drops stale frames), and the sample loop does nothing — costs nothing — while + no browser is connected. + +`VizLauncher.TryStart(args, campaign, env)` is the shared `--viz [port]` handler used by every Lab: it parses the +flag, **requires a Development environment** (else prints a note and skips), checks the campaign is a telemetry +source, and starts the server. Wired into all six games — `--game --viz`. + +### 4.D Static viewer — Angular `/network` (follow-on phase) + +A standalone `network-visualizer` component + lazy route + home card, reusing the existing browser `.ckpt` parsers +to render the same graph/heatmap for a chosen shipped checkpoint — client-side, no server. Shares the visual +language of the live page. + +## 5. Milestone plan + +- **M36.1 — live during training, all games, beginner tooltips + per-neuron live values (this PRD's gate; SHIPPED).** + Pull-based Core seam + every campaign as a source + the async Lab `--viz` viewer with hover tooltips, gated to + Development. Environment-aware labels (Snake, FruitCake, Cube, Rush Hour, DAVI) and **live per-neuron values** + (input feature values, output Q/scores, hidden-neuron activations) — the DQN games forward their real observation, + the batch nets a fixed probe state. **Gate (met):** `--game --viz` → the net visibly evolves in-browser; + tooltips name each neuron and show its live value; a Production process skips the socket; viz vs no-viz checkpoints + are SHA256-identical (verified with a viewer connected). Verified in-browser on Snake, FruitCake, Rush Hour, DAVI. +- **M36.2 — static `.ckpt` inspection in the web app (planned).** Angular `/network` route reusing the browser + `.ckpt` parsers to render the same graph/heatmap for a chosen shipped checkpoint, client-side. +- **M36.3 — richer live view (optional, planned).** Signed (diverging) heatmaps; viewer→trainer controls over the + existing WebSocket (pause/step, cadence, layer-select); gradient-flow overlay; continuous-control PPO/SAC once they + train through the Lab's `--game` dispatch. + +## 6. Risks / watch-items + +- **`HttpListener` prefix permissions.** `http://localhost:/` is allowed for non-admin users on Windows + (unlike `http://+:port/`); keep it localhost-only. +- **Concurrent read race.** Sampling reads weights on a background thread while training writes them — chosen + deliberately (it keeps the seam out of the trainers) and benign for a magnitude heatmap; it is provably harmless to + training output (SHA256-verified). Not suitable if we ever needed a *consistent* snapshot (we don't). +- **Frame rate vs. throughput.** ~150 ms cadence → a few Hz; capture is cheap (stats + ≤24² downsample) and skipped + entirely when no browser is connected. A very large net (DAVI 34 MB) allocates a params snapshot per sampled frame + — fine for a dev tool, and only while someone is watching. +- **Sign is dropped in the heatmap.** v1 shows |w| magnitude (bounded, always positive); signed diverging colour is + a possible M36.3 refinement. +- **GPU-resident nets.** The cube DAVI net's device weights are mirrored by a periodically-synced CPU master; the + viewer reads that master (last-synced weights) — correct and lock-free, just slightly behind the device. + +## 7. Honest ceiling + +This shows **weights** evolving, plus live **input/output values and hidden-neuron activations** for every trainable +game — via a read-only forward per sampled frame. The honest caveat: the batch-trained nets (Cube, Rush Hour, DAVI) +have no running env, so their live values are for a **fixed probe state** (one representative board), not the actual +training batch — so you see how the net's response to that *one* board evolves, which is the point, but it isn't a +random sample of what training currently sees. What it still does **not** show: **gradient** flow (grads aren't +retained at sample time) and **per-connection signed** weights (the heatmap is magnitude-only). Each forward is an +extra pass — negligible for these nets at a few Hz. These are deliberate boundaries; each is a clean follow-up. + +## 8. Sources + +Investigation reports (2026-07-12): checkpoint/model layer, web/WebSocket surface, training-loop hooks, and docs +conventions — summarized inline above. Code references: `Core/Nn/Modules.cs`, `Core/Nn/DuelingQNet.cs`, +`Core/Training/DqnTrainer.cs`, `Core/Training/CampaignRunner.cs`, `tools/…Lab/SnakeDqnCampaign.cs`, +`RLDemo.Web/ClientApp/src/app/*/*-net.ts`, and `../ARCHITECTURE.md` §2/§5/§6/§8. diff --git a/docs/prd/PLAN.md b/docs/prd/PLAN.md index 5ed628c..d631627 100644 --- a/docs/prd/PLAN.md +++ b/docs/prd/PLAN.md @@ -1209,6 +1209,83 @@ Play-yourself identical; steady 60 fps at full board; crisp on hi-DPI; rAF loop Verify by save-and-live-reload against the running host (**do not** run `ng serve`/`ng build`); attach a before/after screenshot/clip to the PR. Single view-only PR. +## M36 — Network visualizer: see the net, and watch it learn *(2026-07-12; branch `m36-network-visualizer` off `master`; see `NETWORK_VISUALIZER_PRD.md`)* ✅ + +**Problem.** A trained net is only ever visible as numbers — a `.ckpt` on disk, a CSV of eval scalars, a console line +per eval. You cannot *see* a network's shape or structure, and — the real gap — you cannot **watch it change as it +trains**. Learning itself is invisible. + +**Fix (M36.1 — shipped; the priority "watch it evolve" gate).** A read-only **pull** telemetry seam plus a viewer +served by the training process itself. A 4-agent investigation (2026-07-12) established the constraint: the web app is +train-free (its server WS infra was removed in M32/M33), so live telemetry must come from the **Lab CLI** where the net +lives; and on the CPU path every parameter is host-resident `float[]`, so reading weights mid-training is free. Design: +- **Core seam** (`Core/Telemetry/NetworkTelemetry.cs`): `INetworkTelemetrySource` (`NetKind` + `SnapshotParameters()` + + `Sample()`) — a **pull** model, so no trainer calls anything. `NetworkInspector` turns any net's parameter tensors + into telemetry by pairing each rank-2 weight with the rank-1 bias after it — keying off `Parameters()` (not a + concrete type) so even the non-`IModule` policy nets work. Frame = per-layer stats + a block-averaged ≤24² magnitude + **heatmap** (payload bounded regardless of width). +- **All six games**: each `ITrainingCampaign` also implements the source in ~4 lines (return the live net's params + + metrics it already tracks) — DQN, imitation-policy, EfficientCube-policy, and DAVI value nets alike. No trainer + changes. Sampling on a background thread is a benign race with training writes and **provably harmless** (no writes/ + RNG/ordering) — **verified**: viz vs no-viz `snake.dqn.ckpt` + `snake.dqn-state.ckpt` are SHA256-identical. +- **Environment-aware tooltips (all games)**: the source optionally exposes `InputLabels`/`OutputLabels` + + `SampleIo()` + `SampleActivations()` (default null → generic). Labels live on the envs — `FruitCakeEnv` (89 + features + 14 columns), `SnakeEnv` (177 egocentric-vision features + 4 directions), `RubiksCubeEnv` (12 + quarter-turns), `RushHourBoard` (32 vehicle×dir moves). **Every neuron shows a live value** in all five games — input feature + values, output Q-values/scores, and **hidden-neuron activations** (each net type exposes `LayerActivations`). The + DQN games forward their actual `CurrentObs`; the batch-trained nets (Cube, Rush Hour, DAVI) forward a **fixed probe + state** (constant-seed scramble / level-1 puzzle), so you watch the net's opinion of one board evolve. Read-only + forwards, CPU for a single row even under the GPU backend → still SHA256-identical with a viewer connected. The + viewer attaches output labels to the column matching their count (a policy net's action head precedes its value + head) and draws labeled columns in full. +- **Live viewer** (`tools/…Lab/VizServer.cs` + shared `VizLauncher.cs`): an `HttpListener` on localhost serving one + self-contained HTML page (Canvas 2D node-link graph + weight-heatmap strip + loss sparkline + **beginner hover + tooltips**; labeled input/output columns drawn in full so each neuron is hoverable) and a **WebSocket** (`GET /ws`). WebSocket (not + SSE) is the owner's call — bidirectional-ready for future viewer→trainer controls. **Fully async** (per-viewer + bounded `Channel` + async send pump + async sample loop; no blocking I/O; costs nothing with no viewer connected). + **Gated to a Development host environment** (`VizLauncher` skips it in Production; the Lab defaults to Development). + `--game --viz [port]` (bare `--viz` = 5250). + +**Gate (met).** `--game snake --viz` → the net **visibly evolves while training** (weight heatmaps/edges shift, +per-layer |w| grows, eval climbs); **hover tooltips** explain each part for RL newcomers; a **Production** process +skips the socket (prints a note) while training proceeds; opening the page mid-run shows the graph instantly; and viz +vs no-viz checkpoints are **SHA256-identical**. 314 fast tests green. Screenshot (with tooltip): +`docs/screenshots/m36-network-visualizer.png`. + +**Follow-ups (planned).** M36.2 — static `.ckpt` inspection as an Angular `/network` page reusing the browser +`.ckpt` parsers (client-side, no server). M36.3 — signed (diverging) heatmaps; viewer→trainer controls over the +existing WebSocket (pause/step, cadence, layer-select); continuous-control PPO/SAC once they train through the Lab's +`--game` dispatch. + +## M37 — Progressive net growth (Net2WiderNet / Net2DeeperNet) *(2026-07-12; branch `m36-network-visualizer`)* ✅ + +**Problem.** A net trains at a fixed architecture; the visualizer made it natural to *watch* a net grow, but the DQN +games had no growth to show. + +**Fix.** Function-preserving architecture growth on the shared `DuelingQNet`: `WidenTo` (Net2WiderNet — new units +duplicate a random existing one, the next layer's incoming weights split evenly across copies) and `Deepen` +(Net2DeeperNet — an extra trunk layer initialized to identity, exact after a ReLU). Both produce a net computing the +**same function** (unit tests assert forward-equality). A shared `DqnGrowth` helper applies a staged schedule +(`[16]→[32]→[32,32]→[64,64]→[64,64,64]→[128,128,128]`, alternating wider/deeper) on a step cadence, rebuilding the +Adam optimizer (moments are keyed to the parameter set) via `DqnTrainingState.WithNetwork` (buffer/RNGs/n-step +accumulator/step-count carried over; obs & action dims unchanged so they stay valid). Wired into **both DQN games**: +`--game snake|fruitcake --grow [--grow-every N]` (starts from the tiny stage, grows mid-run). + +**Coverage — every trainable net now grows.** The growth math is factored into `Net2Net` (`WidenTrunk` + `SetIdentity`) +and shared: +- **DQN games** (Snake, FruitCake) — `DuelingQNet` grows wider+deeper via `--grow` (`DqnGrowth`). +- **Imitation/EfficientCube policy nets** (Cube, Cube-policy, Rush Hour) — the two-headed nets were **refactored** from + a fixed 2-layer trunk to a variable-depth `PolicyValueNet` core (shared by `CubePolicyNet`/`RushHourPolicyNet`). + Checkpoint format bumped to **v2** (stores the trunk-widths array); **v1 shipped files still load** (one hidden width + → a two-layer trunk), guarded by a test. They grow wider+deeper via `--grow` (`PolicyGrowth`, same schedule). +- **DAVI value net** (`ResidualMlp`, cube-davi) — already grew **width** (pre-existing `--auto-widen` / `--grow-to`). + +**Gate (met).** `--game fruitcake --viz --grow` grows the DQN net `[16]`→`[128,128,128]` live (wider **and** deeper), +and `--game rushhour --viz --grow` grows the **policy** net the same way (`docs/screenshots/m37-policy-net-grows.png` — +note the identity diagonals in the just-deepened layers' heatmaps), both with **no loss spike** (function-preserving). +320 fast tests green (incl. Net2Net forward-equality for DuelingQNet + PolicyValueNet, and a v1-checkpoint-load test). +Screenshots: `docs/screenshots/m36-network-grows.png` (DQN), `m37-policy-net-grows.png` (policy). + ## Testing strategy (cross-cutting, from research) 1. **Known-solved thresholds** as integration tests (median over ≥3 seeds) — slow bucket. @@ -1255,6 +1332,8 @@ screenshot/clip to the PR. Single view-only PR. | M28 2048 expectimax (2026-06-26) | beat the shipped n-tuple greedy on score, serving-viable | **~84k vs ~44k (1.9×)**, best tile 4096→8192 over 100 games, *no retrain*; ~1.2 s/playout at depth 1 | | M28 NoisyNets capability (2026-06-26) | learns, serializes, serves deterministically; no shipped checkpoint broken | 293/293; σ-grads flow; noisy resume bitwise; v1 ckpts load as plain; serving unchanged (noise off) | | M28 FruitCake NoisyNets empirical (2026-06-26) | match or beat ε-greedy at equal budget (multi-seed) | **matched** — 200-game paired A/B tie (702.1 vs 714.4, Δ −12.3 ± 29.8 SE); single-evals were seed-luck; not shipped | +| M36.1 network visualizer — watch it train (2026-07-12) | see the net evolve live during training (all games); beginner-readable; zero training impact | **met** — pull-based seam; **all six `--game`s** stream topology + weight frames over a **WebSocket** to a self-contained page with **hover tooltips**; net visibly evolves (heatmaps/edges shift, eval 7.0→13.9); **Development-gated**; viz vs no-viz checkpoints **SHA256-identical**; 314 tests green | +| M37 progressive net growth (2026-07-12) | grow the net wider+deeper mid-training without a loss spike, everywhere possible | **met** — shared `Net2Net` (WidenTrunk/SetIdentity); `--grow` grows **all** trainable nets live: `DuelingQNet` (Snake, FruitCake) and the refactored variable-depth `PolicyValueNet` (Cube, Cube-policy, Rush Hour), `[16]`→`[128,128,128]`; DAVI `ResidualMlp` already grew width. Policy checkpoint → v2 with v1 back-compat (tested). 320 tests (4 new: widen/deepen forward-equality ×2, v1 load, grown round-trip) | ## Shipped (2026-06-11) — release engineering diff --git a/docs/prd/PRD.md b/docs/prd/PRD.md index 603ac04..f2618de 100644 --- a/docs/prd/PRD.md +++ b/docs/prd/PRD.md @@ -589,3 +589,23 @@ oriented head with eyes, cheap multi-pass 3-D shading, and — the biggest singl > body, so the renderer swaps in behind it with **zero change** to `snake-logic.ts` / `snake-director.ts` / > `snake_solver.ts` / the `.pg`. M34's ~81 food@12 strength and all tests are untouched — the gate is a purely visual > in-browser before/after. + +## 17. Network visualizer — see the net, and watch it learn *(PRD §17; inserted 2026-07-12; see [NETWORK_VISUALIZER_PRD.md](NETWORK_VISUALIZER_PRD.md) + [PLAN.md](PLAN.md) M36)* + +A network is otherwise only ever visible as numbers — a `.ckpt` on disk, a CSV of eval scalars. This feature makes a +net **visible**, and in particular lets you **watch it change as it trains** — the priority — and lets a newcomer to +neural nets *read* the picture. It adds a read-only **pull** seam in Core (`INetworkTelemetrySource` + `NetworkInspector`, +which describes any net from its parameter tensors), which every training campaign implements in a few lines, so +`--viz` works for **all six trainable games** with no per-trainer code. The Lab-hosted viewer shows a live node-link +graph + weight heatmaps + a loss sparkline, with **hover tooltips** explaining each neuron/connection in plain +language. Because sampling only *reads* the host-resident parameter arrays (on a background thread), a run being +watched is **bitwise-identical** to one that isn't (verified: viz vs no-viz checkpoints are SHA256-equal). + +> **Design decision (2026-07-12, 4-agent investigation).** The live view is served **by the training process itself**, +> not the web app: `RLDemo.Web` is train-free and training lives in the Lab CLI where the net actually is. So the Lab +> hosts a tiny localhost `HttpListener` + **WebSocket** (`/ws`) feeding one self-contained Canvas 2D page, driven by an +> async background sample loop that pulls the campaign's telemetry source. WebSocket (over one-way SSE) is a deliberate +> call so the channel can become bidirectional — viewer→trainer control (pause/step, cadence, layer-select) — without +> swapping transport. It is **dev-only**: the socket is gated to a Development host environment and is not the web-app +> WS stack removed in M32/M33. Static `.ckpt` inspection instead belongs client-side in the Angular app (it already +> ships `.ckpt` parsers) as a follow-on `/network` page. diff --git a/docs/screenshots/m36-network-grows.png b/docs/screenshots/m36-network-grows.png new file mode 100644 index 0000000..e2820cd Binary files /dev/null and b/docs/screenshots/m36-network-grows.png differ diff --git a/docs/screenshots/m36-network-visualizer.png b/docs/screenshots/m36-network-visualizer.png new file mode 100644 index 0000000..4ef462c Binary files /dev/null and b/docs/screenshots/m36-network-visualizer.png differ diff --git a/docs/screenshots/m37-policy-net-grows.png b/docs/screenshots/m37-policy-net-grows.png new file mode 100644 index 0000000..92ef84d Binary files /dev/null and b/docs/screenshots/m37-policy-net-grows.png differ diff --git a/src/Directory.Build.props b/src/Directory.Build.props index 4bed13c..25cba62 100644 --- a/src/Directory.Build.props +++ b/src/Directory.Build.props @@ -7,7 +7,7 @@ The demo apps (RLDemo.*) under src/ don't declare , so they stay unversioned. --> - 0.3.0 + 0.4.0 diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/DqnTrainingState.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/DqnTrainingState.cs index 6cea517..0592d71 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/DqnTrainingState.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Checkpoints/DqnTrainingState.cs @@ -46,6 +46,30 @@ public sealed class DqnTrainingState /// Opaque environment snapshot; null when the env is not IStatefulEnvironment (resume then re-Resets, losing bitwise equality). public byte[]? EnvState { get; set; } + /// + /// A copy of this state with the network swapped for a (function-preservingly) grown one and a fresh optimizer + /// for its new parameter set — carrying the replay buffer, RNG streams, n-step accumulator, step count and + /// current observation forward unchanged. Used to grow the architecture mid-training (Net2WiderNet/DeeperNet): + /// the observation/action dimensions are unchanged, so the buffer and accumulator stay valid. The optimizer must + /// be new because Adam's moments are keyed to the parameter set. + /// + public DqnTrainingState WithNetwork(IValueNet online, IValueNet target, Adam optimizer) => new() + { + Online = online, + Target = target, + Optimizer = optimizer, + Buffer = Buffer, + PolicyRng = PolicyRng, + BufferRng = BufferRng, + NoiseRng = NoiseRng, + Accumulator = Accumulator, + CurrentObs = CurrentObs, + StepsCompleted = StepsCompleted, + LastLoss = LastLoss, + LastEval = LastEval, + EnvState = EnvState, + }; + public void Save(Stream destination) { using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/DuelingQNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/DuelingQNet.cs index 13dd7d3..5f90daa 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/DuelingQNet.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/DuelingQNet.cs @@ -81,8 +81,76 @@ public IEnumerable Parameters() foreach (var p in _advantageHead.Parameters()) yield return p; } + /// + /// Per-layer activations for a single input row, in layer order: each trunk layer's + /// post-ReLU output, then the value head V(s) and the advantage head A(s,·). Lets a telemetry viewer show every + /// hidden neuron's current value (not just input/output). Read-only — allocates throwaway intermediates and + /// never touches parameters, so it's safe to call while training runs. + /// + public float[][] LayerActivations(Tensor input) + { + var acts = new List(_trunk.Length + 2); + var x = input; + foreach (var layer in _trunk) + { + x = layer.Forward(x).Relu(); + acts.Add([.. x.Data]); + } + acts.Add([.. _valueHead.Forward(x).Data]); + acts.Add([.. _advantageHead.Forward(x).Data]); + return [.. acts]; + } + public IValueNet CloneStructure() => new DuelingQNet(_inputSize, _hidden, _actions, new Xoshiro256StarStar(0), _noisy); + /// The shared-trunk hidden widths — used to drive progressive growth schedules. + public int[] Trunk => [.. _hidden]; + + /// + /// Net2WiderNet (Chen et al. 2016): a wider net that computes the exact same function. Each widened + /// trunk layer's new units duplicate a randomly-chosen existing unit; the next layer's incoming weights from a + /// duplicated unit are split evenly across its copies so every downstream sum is unchanged. Lets training add + /// capacity mid-run without the loss spike a cold-init would cause. Plain (non-noisy) nets only. + /// + public DuelingQNet WidenTo(int[] newHidden, Xoshiro256StarStar rng) + { + if (_noisy) throw new NotSupportedException("WidenTo is only supported for non-noisy DuelingQNet."); + if (newHidden.Length != _hidden.Length) throw new ArgumentException("WidenTo preserves depth; use Deepen to add a layer."); + for (int i = 0; i < _hidden.Length; i++) + if (newHidden[i] < _hidden[i]) throw new ArgumentException("WidenTo cannot shrink a layer."); + + var grown = new DuelingQNet(_inputSize, newHidden, _actions, rng, noisy: false); + Net2Net.WidenTrunk(_inputSize, _trunk, _hidden, grown._trunk, newHidden, + [((Linear)_valueHead, (Linear)grown._valueHead, 1), + ((Linear)_advantageHead, (Linear)grown._advantageHead, _actions)], rng); + return grown; + } + + /// + /// Net2DeeperNet: a deeper net (one extra trunk layer, same width as the last) computing the same + /// function. The inserted layer is initialized to identity (W = I, b = 0); since it follows a ReLU its + /// input is already ≥ 0, so ReLU(I·x) = x. Plain (non-noisy) nets only. + /// + public DuelingQNet Deepen(Xoshiro256StarStar rng) + { + if (_noisy) throw new NotSupportedException("Deepen is only supported for non-noisy DuelingQNet."); + int w = _hidden[^1]; + var newHidden = new int[_hidden.Length + 1]; + Array.Copy(_hidden, newHidden, _hidden.Length); + newHidden[^1] = w; + + var grown = new DuelingQNet(_inputSize, newHidden, _actions, rng, noisy: false); + for (int i = 0; i < _trunk.Length; i++) + { + _trunk[i].Weight.Data.CopyTo(grown._trunk[i].Weight.Data.AsSpan()); + _trunk[i].Bias.Data.CopyTo(grown._trunk[i].Bias.Data.AsSpan()); + } + Net2Net.SetIdentity(grown._trunk[^1]); // new last trunk layer → identity (function-preserving after a ReLU) + Net2Net.CopyLinear((Linear)_valueHead, (Linear)grown._valueHead); + Net2Net.CopyLinear((Linear)_advantageHead, (Linear)grown._advantageHead); + return grown; + } + /// public IValueNet GrowInput(int newInputSize) { diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/Net2Net.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/Net2Net.cs new file mode 100644 index 0000000..0221eb7 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/Net2Net.cs @@ -0,0 +1,95 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Nn; + +/// A net with a plain trunk that can grow function-preservingly (Net2Wider/DeeperNet). Lets one shared +/// growth driver widen/deepen any implementer (the two-headed policy nets) on a schedule. Self-typed so the growth +/// operators return the concrete net type. +public interface IGrowableTrunkNet where TSelf : IGrowableTrunkNet +{ + /// The shared-trunk hidden widths. + int[] Trunk { get; } + /// A wider net (same depth) computing the same function. + TSelf WidenTo(int[] newHidden, Xoshiro256StarStar rng); + /// A deeper net (one extra trunk layer) computing the same function. + TSelf Deepen(Xoshiro256StarStar rng); + IEnumerable Parameters(); +} + +/// +/// Function-preserving architecture-growth primitives (Chen et al. 2016), shared by the nets that grow mid-training +/// (, ): both are a plain ReLU trunk feeding one or more output +/// heads, so the growth math is identical and lives here once. +/// +public static class Net2Net +{ + /// + /// Net2WiderNet: fill (already built with the wider + /// widths) and each output head so the whole net computes the same function. Each widened layer's extra + /// units duplicate a randomly-chosen existing unit; every consumer of a duplicated unit (the next trunk layer, + /// or a head reading the last trunk layer) has its incoming weights from that unit split evenly across the + /// copies, so all downstream sums are unchanged. Heads read the LAST trunk layer's output. + /// + public static void WidenTrunk(int inputSize, Linear[] oldTrunk, int[] oldHidden, Linear[] newTrunk, int[] newHidden, + ReadOnlySpan<(Linear Old, Linear New, int OutDim)> heads, Xoshiro256StarStar rng) + { + int layers = oldTrunk.Length; + var map = new int[layers][]; // new-unit → old-unit + var count = new int[layers][]; // how many new units map to each old unit (the split factor) + for (int i = 0; i < layers; i++) + { + int oldW = oldHidden[i], newW = newHidden[i]; + var g = new int[newW]; + var cnt = new int[oldW]; + for (int j = 0; j < oldW; j++) { g[j] = j; cnt[j] = 1; } + for (int j = oldW; j < newW; j++) { int s = rng.NextInt(oldW); g[j] = s; cnt[s]++; } + map[i] = g; count[i] = cnt; + } + + for (int i = 0; i < layers; i++) + { + Linear oldL = oldTrunk[i], newL = newTrunk[i]; + int oldOut = oldHidden[i], newOut = newHidden[i]; + int newIn = i == 0 ? inputSize : newHidden[i - 1]; + for (int r = 0; r < newIn; r++) + { + int sr = i == 0 ? r : map[i - 1][r]; + float scale = i == 0 ? 1f : 1f / count[i - 1][sr]; + for (int c = 0; c < newOut; c++) + newL.Weight.Data[r * newOut + c] = oldL.Weight.Data[sr * oldOut + map[i][c]] * scale; + } + for (int c = 0; c < newOut; c++) newL.Bias.Data[c] = oldL.Bias.Data[map[i][c]]; + } + + foreach (var (oldHead, newHead, outDim) in heads) + { + int mapLastLen = map[layers - 1].Length; + for (int r = 0; r < mapLastLen; r++) + { + int sr = map[layers - 1][r]; + float scale = 1f / count[layers - 1][sr]; + for (int o = 0; o < outDim; o++) + newHead.Weight.Data[r * outDim + o] = oldHead.Weight.Data[sr * outDim + o] * scale; + } + oldHead.Bias.Data.CopyTo(newHead.Bias.Data.AsSpan()); + } + } + + /// Net2DeeperNet building block: set a square layer to identity (W = I, b = 0). Inserted after a ReLU + /// (whose output is ≥ 0), ReLU(I·x) = x, so it's function-preserving. + public static void SetIdentity(Linear layer) + { + int w = layer.Weight.Cols; + Array.Clear(layer.Weight.Data); + for (int k = 0; k < w; k++) layer.Weight.Data[k * w + k] = 1f; + Array.Clear(layer.Bias.Data); + } + + /// Copy a Linear's weights + bias into an identically-shaped one. + public static void CopyLinear(Linear src, Linear dst) + { + src.Weight.Data.CopyTo(dst.Weight.Data.AsSpan()); + src.Bias.Data.CopyTo(dst.Bias.Data.AsSpan()); + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs new file mode 100644 index 0000000..b1e0220 --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/PolicyValueNet.cs @@ -0,0 +1,147 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Nn; + +/// +/// A two-headed policy/value network: a shared variable-depth ReLU trunk over an observation, a policy head +/// (one logit per action) and a scalar value head. It is the shared core of the imitation policy nets +/// (CubePolicyNet, RushHourPolicyNet), which wrap it to add their environment sizes and checkpoint +/// kind. The trunk is a [] (not a fixed pair) so the net can grow deeper as well as +/// wider mid-training via the function-preserving transforms. +/// +public sealed class PolicyValueNet +{ + private readonly Linear[] _trunk; + private readonly Linear _policyHead, _valueHead; + private readonly int _inputSize, _actions; + + public PolicyValueNet(int inputSize, int[] hidden, int actions, Xoshiro256StarStar rng) + { + if (hidden.Length < 1) throw new ArgumentException("A policy/value net needs at least one trunk layer."); + _inputSize = inputSize; + _actions = actions; + _trunk = new Linear[hidden.Length]; + int prev = inputSize; + for (int i = 0; i < hidden.Length; i++) + { + _trunk[i] = new Linear(prev, hidden[i], rng, Activation.Relu); + prev = hidden[i]; + } + _policyHead = new Linear(prev, actions, rng, Activation.None); + _valueHead = new Linear(prev, 1, rng, Activation.None); + } + + public int InputSize => _inputSize; + public int Actions => _actions; + + /// The shared-trunk hidden widths (drives growth schedules). + public int[] Trunk => [.. _trunk.Select(l => l.Weight.Cols)]; + + /// Batched forward pass (autograd-recorded): raw policy logits [B, actions] + value [B, 1]. + public (Tensor Logits, Tensor Value) Forward(Tensor observations) + { + var x = observations; + foreach (var layer in _trunk) x = layer.Forward(x).Relu(); + return (_policyHead.Forward(x), _valueHead.Forward(x)); + } + + public IEnumerable Parameters() + { + foreach (var layer in _trunk) + foreach (var p in layer.Parameters()) yield return p; + foreach (var p in _policyHead.Parameters()) yield return p; + foreach (var p in _valueHead.Parameters()) yield return p; + } + + /// Per-layer activations for one input row, in order (each trunk layer's + /// post-ReLU output, then the policy head, then the value head) — for a live-network viewer. + public float[][] LayerActivations(Tensor observation) + { + var acts = new List(_trunk.Length + 2); + var x = observation; + foreach (var layer in _trunk) { x = layer.Forward(x).Relu(); acts.Add([.. x.Data]); } + acts.Add([.. _policyHead.Forward(x).Data]); + acts.Add([.. _valueHead.Forward(x).Data]); + return [.. acts]; + } + + /// Net2WiderNet — a wider net computing the same function (see ). + public PolicyValueNet WidenTo(int[] newHidden, Xoshiro256StarStar rng) + { + if (newHidden.Length != _trunk.Length) throw new ArgumentException("WidenTo preserves depth; use Deepen to add a layer."); + for (int i = 0; i < _trunk.Length; i++) + if (newHidden[i] < _trunk[i].Weight.Cols) throw new ArgumentException("WidenTo cannot shrink a layer."); + + var grown = new PolicyValueNet(_inputSize, newHidden, _actions, rng); + Net2Net.WidenTrunk(_inputSize, _trunk, Trunk, grown._trunk, newHidden, + [(_policyHead, grown._policyHead, _actions), (_valueHead, grown._valueHead, 1)], rng); + return grown; + } + + /// Net2DeeperNet — one extra trunk layer (identity-init), same function (see ). + public PolicyValueNet Deepen(Xoshiro256StarStar rng) + { + int w = _trunk[^1].Weight.Cols; + var newHidden = new int[_trunk.Length + 1]; + for (int i = 0; i < _trunk.Length; i++) newHidden[i] = _trunk[i].Weight.Cols; + newHidden[^1] = w; + + var grown = new PolicyValueNet(_inputSize, newHidden, _actions, rng); + for (int i = 0; i < _trunk.Length; i++) Net2Net.CopyLinear(_trunk[i], grown._trunk[i]); + Net2Net.SetIdentity(grown._trunk[^1]); + Net2Net.CopyLinear(_policyHead, grown._policyHead); + Net2Net.CopyLinear(_valueHead, grown._valueHead); + return grown; + } + + /// The policy path (trunk → policy head) as a standalone — for a GPU-resident forward + /// over the policy logits (the value head is irrelevant to search). Weights are COPIED (a frozen snapshot). + public Mlp PolicyAsMlp() + { + int[] sizes = [_inputSize, .. _trunk.Select(l => l.Weight.Cols), _actions]; + var mlp = new Mlp(sizes, new Xoshiro256StarStar(0), Activation.Relu); + for (int i = 0; i < _trunk.Length; i++) Net2Net.CopyLinear(_trunk[i], mlp.Layers[i]); + Net2Net.CopyLinear(_policyHead, mlp.Layers[^1]); + return mlp; + } + + private IEnumerable AllLayers() => [.. _trunk, _policyHead, _valueHead]; + + // ── Checkpoint (kind supplied by the wrapper) ───────────────────────────────────────────────────────────── + // v2: trunk widths (int[]) + every layer's floats. v1 (shipped): a single hidden int → a two-layer trunk. + private const int Version = 2; + + public void Save(Stream destination, string kind) + { + using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); + CheckpointFormat.WriteHeader(writer, kind, Version); + CheckpointFormat.WriteInts(writer, Trunk); + foreach (var layer in AllLayers()) + { + CheckpointFormat.WriteFloats(writer, layer.Weight.Data); + CheckpointFormat.WriteFloats(writer, layer.Bias.Data); + } + } + + /// Loads a policy/value net. / come from the + /// environment (they were never stored in the file); the trunk shape comes from the file (v1: one hidden width → + /// two layers; v2: an explicit widths array), so grown checkpoints round-trip and shipped v1 files still load. + public static PolicyValueNet Load(Stream source, string kind, int inputSize, int actions) + { + using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true); + int version = CheckpointFormat.ReadHeader(reader, kind, Version); + int[] hidden = version >= 2 ? CheckpointFormat.ReadInts(reader) : [reader.ReadInt32(), 0]; + if (version < 2) hidden[1] = hidden[0]; // v1 stored one hidden width for a fixed two-layer trunk + + var net = new PolicyValueNet(inputSize, hidden, actions, new Xoshiro256StarStar(0)); + foreach (var layer in net.AllLayers()) + { + CheckpointFormat.ReadFloats(reader).CopyTo(layer.Weight.Data.AsSpan()); + CheckpointFormat.ReadFloats(reader).CopyTo(layer.Bias.Data.AsSpan()); + } + return net; + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ResidualMlp.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ResidualMlp.cs index 8522de3..0c99663 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ResidualMlp.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Nn/ResidualMlp.cs @@ -79,6 +79,28 @@ public Tensor Forward(Tensor input) return _head.Forward(x); } + /// + /// Per-layer activations for one input row, aligned to the fully-connected layers a telemetry viewer recovers + /// from (input projection, then each block's two Linears, then the head): the + /// projected+normed trunk, and per block the inner ReLU output and the post-skip residual sum. Read-only — + /// allocates throwaway intermediates, so it's safe to call while training runs. + /// + public float[][] LayerActivations(Tensor input) + { + var acts = new List(2 + 2 * _block1.Length); + var x = _inProj.Forward(input).LayerNorm(_inGamma, _inBeta).Relu(); + acts.Add([.. x.Data]); + for (int i = 0; i < _block1.Length; i++) + { + var h = _block1[i].Forward(x).LayerNorm(_blockGamma[i], _blockBeta[i]).Relu(); + acts.Add([.. h.Data]); + x = x.Add(_block2[i].Forward(h)); // residual skip + acts.Add([.. x.Data]); + } + acts.Add([.. _head.Forward(x).Data]); + return [.. acts]; + } + public IEnumerable Parameters() { foreach (var p in _inProj.Parameters()) yield return p; diff --git a/src/MintPlayer.AI.ReinforcementLearning.Core/Telemetry/NetworkTelemetry.cs b/src/MintPlayer.AI.ReinforcementLearning.Core/Telemetry/NetworkTelemetry.cs new file mode 100644 index 0000000..1cd24fa --- /dev/null +++ b/src/MintPlayer.AI.ReinforcementLearning.Core/Telemetry/NetworkTelemetry.cs @@ -0,0 +1,194 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; + +namespace MintPlayer.AI.ReinforcementLearning.Core.Telemetry; + +/// +/// The read-only telemetry seam a training campaign exposes so an out-of-process viewer can watch its network +/// evolve as it learns. It is a pull model: a campaign publishes its live parameter tensors plus the run's +/// scalar metrics, and a viewer samples them on its own cadence. Working from the parameter tensors — not a +/// concrete net type — means every architecture is covered uniformly (the DQN DuelingQNet/Mlp, the +/// two-headed policy nets, the DAVI ResidualMlp), and no trainer needs to know telemetry exists. +/// +/// Sampling is a pure observation of parameter : it never mutates the net, the RNG, or +/// the optimizer, so a run being watched trains identically to one that isn't. On the CPU path the arrays are +/// host-resident, so a read is free; it races the training thread's writes, but a torn value only affects one +/// pixel of a magnitude heatmap — cosmetically irrelevant, and never a fault. +/// +/// +public interface INetworkTelemetrySource +{ + /// A free-form label for the network kind (e.g. "dueling-q", "cube-policy", "value-davi-res"). + string NetKind { get; } + + /// The net's current parameter tensors (weights + biases, in forward order), or null until the net + /// exists (training hasn't started / resumed yet). Returns a snapshot list so the caller can iterate safely. + IReadOnlyList? SnapshotParameters(); + + /// The run's scalar metrics at this instant (step, loss, eval, …). Cheap; called once per frame. + NetworkMetrics Sample(); + + // ── Optional semantics (default: none → the viewer shows generic tooltips). An environment that knows what its + // observation features and actions MEAN can override these so a viewer can say, per neuron, what it represents + // and — via SampleIo — its current value. Static labels live on the topology; live values on each frame. ── + + /// Human-readable name for each INPUT neuron (length = input width), or null if unknown. + IReadOnlyList? InputLabels => null; + + /// Human-readable name for each OUTPUT neuron / action (length = output width), or null if unknown. + IReadOnlyList? OutputLabels => null; + + /// The net's current input vector and the output it produces for it (e.g. per-action Q-values), so a + /// viewer can show "what this input is right now" and "what each output computes". Null when unavailable. + (float[] Input, float[] Output)? SampleIo() => null; + + /// Each layer's current activation vector (post-activation output), in the same order as the layers + /// recovers — so a viewer can show every hidden neuron's live value, not just the + /// input/output. Null when the net can't (cheaply) expose its intermediate activations. + float[][]? SampleActivations() => null; +} + +/// The run's scalar read-outs at a moment in training. Any field may be when a +/// metric doesn't apply to this algorithm (e.g. ε for a supervised campaign) — the viewer renders NaN as "—". +public readonly record struct NetworkMetrics(long Step, long MaxSteps, double Loss, double Eval, double Epsilon); + +/// One fully-connected layer's shape. neurons fed by an +/// [×] weight matrix; is a +/// display hint ("hidden" for trunk layers, "output" for the final head). +public sealed record LayerInfo(int Index, int InputSize, int OutputSize, string Role); + +/// A network's structure: the input width and the ordered fully-connected layers recovered from the +/// parameter tensors. Stable for the whole run — the viewer lays the graph out from this once. +/// / name each input/output neuron when the +/// environment supplies them (null otherwise). +public sealed record NetworkTopology( + string NetKind, int InputSize, int OutputSize, IReadOnlyList Layers, + IReadOnlyList? InputLabels = null, IReadOnlyList? OutputLabels = null); + +/// +/// One layer's weights at a moment in training: summary stats plus a downsampled magnitude heatmap +/// ( is × block-averaged |weight|, row-major) so a frame +/// stays small regardless of the true matrix size (a 1024×1024 layer streams the same handful of KB as a 16×16). +/// +public sealed record LayerFrame( + int Index, int Rows, int Cols, + float WMin, float WMax, float WMeanAbs, float WL2, + float BiasMeanAbs, + int HRows, int HCols, float[] Heat); + +/// A whole-network snapshot: the training scalars at this step plus every layer's +/// , and (when the source supplies them) the net's current input vector and the output it +/// produces — so a viewer can show each input/output neuron's live value. +public sealed record NetworkFrame( + long Step, long MaxSteps, + double Loss, double Eval, double Epsilon, + IReadOnlyList Layers, + float[]? InputValues = null, float[]? OutputValues = null, + float[][]? Activations = null); + +/// +/// Turns a net's parameter tensors into telemetry. It pairs each rank-2 weight tensor with the rank-1 bias that +/// follows it into a fully-connected layer — the layout every net in this library emits — so the viewer needs no +/// per-architecture knowledge. Weight matrices are row-major [in, out] (Data[r*Cols + c]), matching +/// . (Parallel heads — a dueling net's value/advantage, a policy net's policy/value — are +/// rendered as successive columns; a benign cosmetic simplification, since they share the trunk output.) +/// +public static class NetworkInspector +{ + /// The (weight, bias) layers among , in forward order. + public static IReadOnlyList<(Tensor Weight, Tensor Bias)> Layers(IReadOnlyList parameters) + { + var layers = new List<(Tensor, Tensor)>(); + Tensor? pendingWeight = null; + foreach (var p in parameters) + { + if (p.Rank == 2) pendingWeight = p; + else if (p.Rank == 1 && pendingWeight is not null) { layers.Add((pendingWeight, p)); pendingWeight = null; } + } + return layers; + } + + /// The net's fixed structure. is a free-form label (e.g. "dueling-q"); + /// / name the input/output neurons when known. + public static NetworkTopology Describe(IReadOnlyList parameters, string netKind, + IReadOnlyList? inputLabels = null, IReadOnlyList? outputLabels = null) + { + var layers = Layers(parameters); + var infos = new LayerInfo[layers.Count]; + for (int i = 0; i < layers.Count; i++) + { + var w = layers[i].Weight; + infos[i] = new LayerInfo(i, w.Rows, w.Cols, i == layers.Count - 1 ? "output" : "hidden"); + } + int inputSize = layers.Count > 0 ? layers[0].Weight.Rows : 0; + int outputSize = layers.Count > 0 ? layers[^1].Weight.Cols : 0; + // Pass labels through as-is: the viewer attaches input labels to the input column and output labels to + // whichever column matches their count. (A multi-head net's action head isn't the final column — a scalar + // value head follows it — so we deliberately do NOT force output labels to match the last layer here.) + return new NetworkTopology(netKind, inputSize, outputSize, infos, inputLabels, outputLabels); + } + + /// + /// Captures the current weights as a carrying the supplied . + /// Each weight matrix is block-averaged (of |value|) down to at most ² cells so the + /// frame size is bounded no matter how wide the layer is. + /// + public static NetworkFrame CaptureFrame(IReadOnlyList parameters, NetworkMetrics metrics, + float[]? inputValues = null, float[]? outputValues = null, float[][]? activations = null, int maxHeat = 24) + { + var layers = Layers(parameters); + var frames = new LayerFrame[layers.Count]; + for (int i = 0; i < layers.Count; i++) + { + var (weight, bias) = layers[i]; + int rows = weight.Rows, cols = weight.Cols; + float[] w = weight.Data; + + float min = float.PositiveInfinity, max = float.NegativeInfinity; + double sumAbs = 0, sumSq = 0; + for (int k = 0; k < w.Length; k++) + { + float v = w[k]; + if (v < min) min = v; + if (v > max) max = v; + sumAbs += Math.Abs(v); + sumSq += (double)v * v; + } + float meanAbs = w.Length == 0 ? 0 : (float)(sumAbs / w.Length); + float l2 = (float)Math.Sqrt(sumSq); + + double biasSumAbs = 0; + foreach (float b in bias.Data) biasSumAbs += Math.Abs(b); + float biasMeanAbs = bias.Length == 0 ? 0 : (float)(biasSumAbs / bias.Length); + + var (heat, hRows, hCols) = Downsample(w, rows, cols, maxHeat); + frames[i] = new LayerFrame(i, rows, cols, min, max, meanAbs, l2, biasMeanAbs, hRows, hCols, heat); + } + return new NetworkFrame(metrics.Step, metrics.MaxSteps, metrics.Loss, metrics.Eval, metrics.Epsilon, frames, + inputValues, outputValues, activations); + } + + /// Block-mean of |weight| onto an at-most maxHeat² grid (row-major), preserving aspect within the cap. + private static (float[] Heat, int HRows, int HCols) Downsample(float[] w, int rows, int cols, int maxHeat) + { + int hRows = Math.Min(rows, maxHeat); + int hCols = Math.Min(cols, maxHeat); + if (hRows <= 0 || hCols <= 0) return ([], 0, 0); + + var heat = new float[hRows * hCols]; + var counts = new int[hRows * hCols]; + for (int r = 0; r < rows; r++) + { + int hr = r * hRows / rows; + for (int c = 0; c < cols; c++) + { + int hc = c * hCols / cols; + int idx = hr * hCols + hc; + heat[idx] += Math.Abs(w[r * cols + c]); + counts[idx]++; + } + } + for (int k = 0; k < heat.Length; k++) + if (counts[k] > 0) heat[k] /= counts[k]; + return (heat, hRows, hCols); + } +} diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/FruitCake/FruitCakeEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/FruitCake/FruitCakeEnv.cs index 6af8a2d..05fb159 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/FruitCake/FruitCakeEnv.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/FruitCake/FruitCakeEnv.cs @@ -1,3 +1,4 @@ +using System.Linq; using System.Text; using MintPlayer.AI.ReinforcementLearning.Core.Environments; using MintPlayer.AI.ReinforcementLearning.Core.Random; @@ -30,6 +31,41 @@ public sealed class FruitCakeEnv : IEnvironment, IStatefulEnvironm // positions the bare skyline collapses, so the net can locate where the (possibly buried) biggest fruit sits. public const int ObservationSize = ColumnCount * 5 + 5 + 5 + 3 + 6; // 89 + /// Plain-language name for each of the observation features, in index + /// order (mirrors fruitcake_solver.pg buildObservation). Lets a visualizer say what each input + /// neuron actually represents. Built once; safe to share. + public static readonly IReadOnlyList ObservationLabels = BuildObservationLabels(); + + /// Plain-language name for each of the actions (drop columns). + public static readonly IReadOnlyList ActionLabels = + [.. Enumerable.Range(0, ColumnCount).Select(c => $"Drop the current fruit in column {c + 1} of {ColumnCount}")]; + + private static string[] BuildObservationLabels() + { + var labels = new List(ObservationSize); + // Block A — per column, interleaved (surface height, top-fruit tier). + for (int c = 0; c < ColumnCount; c++) + { + labels.Add($"Column {c + 1}: surface height (0 = empty … 1 = piled to the top)"); + labels.Add($"Column {c + 1}: tier of the fruit on top (÷11)"); + } + for (int c = 0; c < ColumnCount; c++) labels.Add($"Column {c + 1}: danger margin (1 = safe, 0 = at the danger line)"); + for (int c = 0; c < ColumnCount; c++) labels.Add($"Column {c + 1}: dropping the current fruit here merges immediately (1/0)"); + for (int c = 0; c < ColumnCount; c++) labels.Add($"Column {c + 1}: the top fruit already has a same-tier neighbour (1/0)"); + for (int t = 1; t <= FruitCatalog.MaxDroppableTier; t++) labels.Add($"Current fruit is tier {t} (one-hot)"); + for (int t = 1; t <= FruitCatalog.MaxDroppableTier; t++) labels.Add($"Next fruit is tier {t} (one-hot)"); + labels.Add("Fruit count on the board (÷100)"); + labels.Add("Board fill ratio (area covered)"); + labels.Add("Highest surface overall (0 = pile at the very top)"); + labels.Add("Biggest fruit: x position (÷ width)"); + labels.Add("Biggest fruit: y position (÷ height)"); + labels.Add("Biggest fruit: tier (÷11)"); + labels.Add("2nd-biggest fruit: x position (÷ width)"); + labels.Add("2nd-biggest fruit: y position (÷ height)"); + labels.Add("2nd-biggest fruit: tier (÷11)"); + return [.. labels]; + } + private const float RewardScale = 10f; // normalize merge points private const float TerminalPenalty = -1f; public const float SettleSpeedPx = 30f; // early-settle: below this (and no merge) the drop is at rest diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/CubePolicyNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/CubePolicyNet.cs index 241a1f5..59d596c 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/CubePolicyNet.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/CubePolicyNet.cs @@ -1,5 +1,3 @@ -using System.Text; -using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Nn; using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Random; @@ -7,42 +5,52 @@ namespace MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; /// -/// Two-headed policy/value network for cube imitation learning (PLAN M16), mirroring -/// : a shared ReLU trunk over the one-hot -/// sticker observation, a 12-way quarter-turn policy head and a scalar value head -/// predicting quarter-turn distance-to-solved (normalized by ). -/// The value head doubles as the heuristic for policy-guided A*. +/// Two-headed policy/value network for cube imitation / EfficientCube learning: a shared ReLU trunk over the +/// one-hot sticker observation, a 12-way quarter-turn policy head and a scalar value head predicting quarter-turn +/// distance-to-solved. A thin wrapper over the shared that fixes the cube's +/// observation/action sizes and checkpoint kind; the trunk is variable-depth, so the net can grow wider +/// () and deeper () mid-training (Net2Net). /// -public sealed class CubePolicyNet +public sealed class CubePolicyNet : IGrowableTrunkNet { public const string CheckpointKind = "cube-policy"; - private const int Version = 1; public const float DistanceScale = 30f; - private readonly Linear _trunk1, _trunk2, _policyHead, _valueHead; + private readonly PolicyValueNet _core; + /// A fresh net with the classic two-layer trunk of the given width. public CubePolicyNet(Xoshiro256StarStar rng, int hidden = 512) - { - _trunk1 = new Linear(RubiksCubeEnv.ObservationSize, hidden, rng, Activation.Relu); - _trunk2 = new Linear(hidden, hidden, rng, Activation.Relu); - _policyHead = new Linear(hidden, RubiksCubeEnv.ActionCount, rng, Activation.None); - _valueHead = new Linear(hidden, 1, rng, Activation.None); - } + : this(new PolicyValueNet(RubiksCubeEnv.ObservationSize, [hidden, hidden], RubiksCubeEnv.ActionCount, rng)) { } + + /// A fresh net with an explicit trunk shape (e.g. a small stage for a growing run). + public CubePolicyNet(Xoshiro256StarStar rng, int[] trunk) + : this(new PolicyValueNet(RubiksCubeEnv.ObservationSize, trunk, RubiksCubeEnv.ActionCount, rng)) { } + + private CubePolicyNet(PolicyValueNet core) => _core = core; - public IEnumerable Parameters() - => _trunk1.Parameters().Concat(_trunk2.Parameters()) - .Concat(_policyHead.Parameters()).Concat(_valueHead.Parameters()); + /// The shared-trunk hidden widths (drives growth schedules). + public int[] Trunk => _core.Trunk; + + public IEnumerable Parameters() => _core.Parameters(); /// Batched forward pass (autograd-recorded): raw policy logits [B,12] + value [B,1]. - public (Tensor Logits, Tensor Value) Forward(Tensor observations) - { - var h = _trunk2.Forward(_trunk1.Forward(observations).Relu()).Relu(); - return (_policyHead.Forward(h), _valueHead.Forward(h)); - } + public (Tensor Logits, Tensor Value) Forward(Tensor observations) => _core.Forward(observations); + + /// Per-layer activations for one input row (for the live-network viewer). + public float[][] LayerActivations(Tensor observation) => _core.LayerActivations(observation); + + /// Net2WiderNet: a wider net computing the same function. + public CubePolicyNet WidenTo(int[] newHidden, Xoshiro256StarStar rng) => new(_core.WidenTo(newHidden, rng)); + + /// Net2DeeperNet: a deeper net (one extra trunk layer) computing the same function. + public CubePolicyNet Deepen(Xoshiro256StarStar rng) => new(_core.Deepen(rng)); + + /// The policy path as a standalone for a GPU-resident beam-search forward. + public Mlp PolicyAsMlp() => _core.PolicyAsMlp(); /// - /// Single-state inference: logits (the inverse of masked - /// to −∞, −1 = none) and predicted distance-to-solved in quarter-turn MOVES. + /// Single-state inference: logits (the inverse of masked to −∞, −1 = none) and + /// predicted distance-to-solved in quarter-turn MOVES. /// public (float[] Logits, float Distance) Evaluate(FaceletCube cube, int lastAction = -1) { @@ -51,7 +59,7 @@ public IEnumerable Parameters() using (GradMode.NoGrad()) { - var (logits, value) = Forward(new Tensor(obs, 1, obs.Length)); + var (logits, value) = _core.Forward(new Tensor(obs, 1, obs.Length)); var masked = new float[RubiksCubeEnv.ActionCount]; int undo = RubiksCubeEnv.InverseAction(lastAction); for (int a = 0; a < masked.Length; a++) @@ -60,50 +68,8 @@ public IEnumerable Parameters() } } - /// - /// The policy path (trunk → trunk → policy head) as a standalone , so a GPU backend - /// can build a device-resident forward over it (the value head is irrelevant to beam search). Weights - /// are COPIED, so the result is a frozen snapshot — rebuild it after training to pick up new weights. - /// - public Mlp PolicyAsMlp() - { - int hidden = _trunk1.Weight.Cols; - var mlp = new Mlp([RubiksCubeEnv.ObservationSize, hidden, hidden, RubiksCubeEnv.ActionCount], - new Xoshiro256StarStar(0), Activation.Relu); - var source = new[] { _trunk1, _trunk2, _policyHead }; - for (int i = 0; i < source.Length; i++) - { - source[i].Weight.Data.CopyTo(mlp.Layers[i].Weight.Data.AsSpan()); - source[i].Bias.Data.CopyTo(mlp.Layers[i].Bias.Data.AsSpan()); - } - return mlp; - } - - public void Save(Stream destination) - { - using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); - CheckpointFormat.WriteHeader(writer, CheckpointKind, Version); - writer.Write(_trunk1.Weight.Cols); // hidden size - foreach (var layer in Layers()) - { - CheckpointFormat.WriteFloats(writer, layer.Weight.Data); - CheckpointFormat.WriteFloats(writer, layer.Bias.Data); - } - } + public void Save(Stream destination) => _core.Save(destination, CheckpointKind); public static CubePolicyNet Load(Stream source) - { - using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true); - CheckpointFormat.ReadHeader(reader, CheckpointKind, Version); - int hidden = reader.ReadInt32(); - var net = new CubePolicyNet(new Xoshiro256StarStar(0), hidden); - foreach (var layer in net.Layers()) - { - CheckpointFormat.ReadFloats(reader).CopyTo(layer.Weight.Data.AsSpan()); - CheckpointFormat.ReadFloats(reader).CopyTo(layer.Bias.Data.AsSpan()); - } - return net; - } - - private IEnumerable Layers() => [_trunk1, _trunk2, _policyHead, _valueHead]; + => new(PolicyValueNet.Load(source, CheckpointKind, RubiksCubeEnv.ObservationSize, RubiksCubeEnv.ActionCount)); } diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/RubiksCubeEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/RubiksCubeEnv.cs index 97f5840..bf4f49f 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/RubiksCubeEnv.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/RubiksCube/RubiksCubeEnv.cs @@ -20,6 +20,25 @@ public sealed class RubiksCubeEnv : IEnvironment, IActionMaskProvi public const int ObservationSize = FaceletCube.FaceletCount * FaceletCube.FaceCount; // 324 public const int ActionCount = 12; + /// Plain-language name for each of the 12 quarter-turn actions (index = action id), e.g. + /// "R' — turn the Right face counter-clockwise". Lets a visualizer label the policy head's outputs. + public static readonly IReadOnlyList ActionLabels = BuildActionLabels(); + + private static string[] BuildActionLabels() + { + var faceName = new Dictionary + { ['U'] = "Up", ['D'] = "Down", ['L'] = "Left", ['R'] = "Right", ['F'] = "Front", ['B'] = "Back" }; + var moves = FaceletCube.QuarterTurnMoves; + var labels = new string[moves.Length]; + for (int i = 0; i < moves.Length; i++) + { + string m = moves[i]; + bool ccw = m.Length > 1 && m[1] == '\''; + labels[i] = $"{m} — turn the {faceName[m[0]]} face {(ccw ? "counter-clockwise" : "clockwise")}"; + } + return labels; + } + private readonly int _maxScrambleDepth; private readonly int _maxMoves; private Xoshiro256StarStar _rng = new(0); diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHour.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHour.cs index 17eec16..2c932ae 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHour.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHour.cs @@ -59,6 +59,19 @@ public static class RushHourBoard public const int MaxVehicles = 16; public const int ActionCount = MaxVehicles * 2; + /// Plain-language name for each of the actions (index = vehicle·2 + dir, + /// dir 0 = toward smaller coords). Vehicle 0 is the red car. Lets a visualizer label the policy head's outputs. + public static readonly IReadOnlyList ActionLabels = + [ + .. Enumerable.Range(0, ActionCount).Select(a => + { + int v = a / 2, dir = a % 2; + string who = v == 0 ? "the red car (vehicle 0)" : $"vehicle {v}"; + string d = dir == 0 ? "up / left (toward the top-left)" : "down / right (toward the bottom-right)"; + return $"Move {who} one cell {d}"; + }) + ]; + public static int[] InitialPositions(RushHourPuzzle puzzle) => [.. puzzle.Vehicles.Select(v => v.Horizontal ? v.Col : v.Row)]; diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHourPolicyNet.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHourPolicyNet.cs index d8e1120..9a85bce 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHourPolicyNet.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/RushHour/RushHourPolicyNet.cs @@ -1,5 +1,3 @@ -using System.Text; -using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Nn; using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Random; @@ -7,37 +5,45 @@ namespace MintPlayer.AI.ReinforcementLearning.Environments.RushHour; /// -/// Two-headed policy/value network for Rush Hour imitation learning: a shared ReLU -/// trunk, a 32-way policy head (one logit per masked action) and a scalar value head -/// predicting distance-to-goal (normalized by ). The value -/// head doubles as the heuristic for policy-guided A* search. +/// Two-headed policy/value network for Rush Hour imitation learning: a shared ReLU trunk, a 32-way policy head +/// (one logit per masked action) and a scalar value head predicting distance-to-goal. A thin wrapper over the +/// shared fixing Rush Hour's observation/action sizes and checkpoint kind; the trunk +/// is variable-depth, so the net can grow wider () and deeper () +/// mid-training (Net2Net). /// -public sealed class RushHourPolicyNet +public sealed class RushHourPolicyNet : IGrowableTrunkNet { public const string CheckpointKind = "rushhour-policy"; - private const int Version = 1; public const float DistanceScale = 20f; - private readonly Linear _trunk1, _trunk2, _policyHead, _valueHead; + private readonly PolicyValueNet _core; + /// A fresh net with the classic two-layer trunk of the given width. public RushHourPolicyNet(Xoshiro256StarStar rng, int hidden = 384) - { - _trunk1 = new Linear(RushHourBoard.ObservationSize, hidden, rng, Activation.Relu); - _trunk2 = new Linear(hidden, hidden, rng, Activation.Relu); - _policyHead = new Linear(hidden, RushHourBoard.ActionCount, rng, Activation.None); - _valueHead = new Linear(hidden, 1, rng, Activation.None); - } + : this(new PolicyValueNet(RushHourBoard.ObservationSize, [hidden, hidden], RushHourBoard.ActionCount, rng)) { } + + /// A fresh net with an explicit trunk shape (e.g. a small stage for a growing run). + public RushHourPolicyNet(Xoshiro256StarStar rng, int[] trunk) + : this(new PolicyValueNet(RushHourBoard.ObservationSize, trunk, RushHourBoard.ActionCount, rng)) { } + + private RushHourPolicyNet(PolicyValueNet core) => _core = core; + + /// The shared-trunk hidden widths (drives growth schedules). + public int[] Trunk => _core.Trunk; - public IEnumerable Parameters() - => _trunk1.Parameters().Concat(_trunk2.Parameters()) - .Concat(_policyHead.Parameters()).Concat(_valueHead.Parameters()); + public IEnumerable Parameters() => _core.Parameters(); /// Batched forward pass (autograd-recorded): raw policy logits [B,32] + value [B,1]. - public (Tensor Logits, Tensor Value) Forward(Tensor observations) - { - var h = _trunk2.Forward(_trunk1.Forward(observations).Relu()).Relu(); - return (_policyHead.Forward(h), _valueHead.Forward(h)); - } + public (Tensor Logits, Tensor Value) Forward(Tensor observations) => _core.Forward(observations); + + /// Per-layer activations for one input row (for the live-network viewer). + public float[][] LayerActivations(Tensor observation) => _core.LayerActivations(observation); + + /// Net2WiderNet: a wider net computing the same function. + public RushHourPolicyNet WidenTo(int[] newHidden, Xoshiro256StarStar rng) => new(_core.WidenTo(newHidden, rng)); + + /// Net2DeeperNet: a deeper net (one extra trunk layer) computing the same function. + public RushHourPolicyNet Deepen(Xoshiro256StarStar rng) => new(_core.Deepen(rng)); /// Single-state inference: masked logits (illegal = −∞) and predicted distance-to-goal in MOVES. public (float[] Logits, float Distance) Evaluate(RushHourPuzzle puzzle, ReadOnlySpan positions) @@ -48,7 +54,7 @@ public IEnumerable Parameters() using (GradMode.NoGrad()) { - var (logits, value) = Forward(new Tensor(obs, 1, obs.Length)); + var (logits, value) = _core.Forward(new Tensor(obs, 1, obs.Length)); var masked = new float[RushHourBoard.ActionCount]; for (int a = 0; a < masked.Length; a++) masked[a] = mask[a] ? logits.Data[a] : float.NegativeInfinity; @@ -56,31 +62,8 @@ public IEnumerable Parameters() } } - public void Save(Stream destination) - { - using var writer = new BinaryWriter(destination, Encoding.UTF8, leaveOpen: true); - CheckpointFormat.WriteHeader(writer, CheckpointKind, Version); - writer.Write(_trunk1.Weight.Cols); // hidden size - foreach (var layer in Layers()) - { - CheckpointFormat.WriteFloats(writer, layer.Weight.Data); - CheckpointFormat.WriteFloats(writer, layer.Bias.Data); - } - } + public void Save(Stream destination) => _core.Save(destination, CheckpointKind); public static RushHourPolicyNet Load(Stream source) - { - using var reader = new BinaryReader(source, Encoding.UTF8, leaveOpen: true); - CheckpointFormat.ReadHeader(reader, CheckpointKind, Version); - int hidden = reader.ReadInt32(); - var net = new RushHourPolicyNet(new Xoshiro256StarStar(0), hidden); - foreach (var layer in net.Layers()) - { - CheckpointFormat.ReadFloats(reader).CopyTo(layer.Weight.Data.AsSpan()); - CheckpointFormat.ReadFloats(reader).CopyTo(layer.Bias.Data.AsSpan()); - } - return net; - } - - private IEnumerable Layers() => [_trunk1, _trunk2, _policyHead, _valueHead]; + => new(PolicyValueNet.Load(source, CheckpointKind, RushHourBoard.ObservationSize, RushHourBoard.ActionCount)); } diff --git a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs index 6ebd011..fbd7a77 100644 --- a/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs +++ b/src/MintPlayer.AI.ReinforcementLearning.Environments/Snake/SnakeEnv.cs @@ -27,6 +27,53 @@ public sealed class SnakeEnv : IEnvironment, IActionMaskProvider, public const int ScalarFeatures = 15; // foodΔ(2)+dist(1)+heading(4)+len(1)+free(4)+tailΔ(2)+tailDist(1) public const int ObservationSize = PatchSize + ScalarFeatures; // 177 + /// Plain-language name for each of the 4 actions (absolute directions), index = action id. + public static readonly IReadOnlyList ActionLabels = ["Move Up", "Move Down", "Move Left", "Move Right"]; + + /// Plain-language name for each of the observation features, in index + /// order (mirrors snake_solver.pg buildObservation): a 9×9 egocentric obstacle plane, a 9×9 food + /// plane, then the scalar block. Lets a visualizer say what each input neuron represents. + public static readonly IReadOnlyList ObservationLabels = BuildObservationLabels(); + + private static string[] BuildObservationLabels() + { + const int plane = PatchSide * PatchSide; // 81 + var labels = new string[ObservationSize]; + for (int i = 0; i < plane; i++) + { + int dr = i / PatchSide - PatchRadius, dc = i % PatchSide - PatchRadius; + string where = PatchWhere(dr, dc); + labels[i] = $"Vision: is a wall/body {where}? (egocentric, head-relative)"; + labels[plane + i] = $"Vision: is the food {where}? (egocentric, head-relative)"; + } + int s = PatchSize; + labels[s++] = "Food offset: columns right of the head (− = left)"; + labels[s++] = "Food offset: rows below the head (− = above)"; + labels[s++] = "Distance to the food (normalized)"; + labels[s++] = "Current heading is Up"; + labels[s++] = "Current heading is Down"; + labels[s++] = "Current heading is Left"; + labels[s++] = "Current heading is Right"; + labels[s++] = "Snake length (÷ board cells)"; + labels[s++] = "Reachable free space if it moves Up (÷ cells)"; + labels[s++] = "Reachable free space if it moves Down (÷ cells)"; + labels[s++] = "Reachable free space if it moves Left (÷ cells)"; + labels[s++] = "Reachable free space if it moves Right (÷ cells)"; + labels[s++] = "Tail offset: columns right of the head (− = left)"; + labels[s++] = "Tail offset: rows below the head (− = above)"; + labels[s++] = "Distance to the tail (normalized)"; + return labels; + } + + // Describe an egocentric patch cell's offset from the head, e.g. (-3, 2) → "3 up, 2 right". + private static string PatchWhere(int dr, int dc) + { + if (dr == 0 && dc == 0) return "at the head"; + string? v = dr < 0 ? $"{-dr} up" : dr > 0 ? $"{dr} down" : null; + string? h = dc < 0 ? $"{-dc} left" : dc > 0 ? $"{dc} right" : null; + return v is null ? h! : h is null ? v : $"{v}, {h}"; + } + public const float FoodReward = 1f; public const float StepPenalty = -0.01f; public const float DeathReward = -1f; diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/DuelingDqnTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/DuelingDqnTests.cs index 372655b..597ad26 100644 --- a/tests/MintPlayer.AI.ReinforcementLearning.Tests/DuelingDqnTests.cs +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/DuelingDqnTests.cs @@ -61,6 +61,32 @@ public void DuelingQNetCheckpoint_RoundTrip_PreservesForward() Assert.Equal(net.Forward(x).Data, loaded.Forward(x).Data); } + [Fact] + public void DuelingQNet_WidenTo_IsFunctionPreserving() + { + var net = new DuelingQNet(6, [10, 14], 4, new Xoshiro256StarStar(9)); + var x = new Tensor([0.4f, -0.2f, 0.1f, 0.3f, -0.5f, 0.2f], 1, 6); + var before = (float[])net.Forward(x).Data.Clone(); + + var wider = net.WidenTo([20, 30], new Xoshiro256StarStar(3)); // Net2WiderNet + Assert.Equal([20, 30], wider.HiddenSizes); + var after = wider.Forward(x).Data; + for (int a = 0; a < before.Length; a++) Assert.Equal(before[a], after[a], 3); + } + + [Fact] + public void DuelingQNet_Deepen_IsFunctionPreserving() + { + var net = new DuelingQNet(6, [12, 12], 4, new Xoshiro256StarStar(11)); + var x = new Tensor([0.2f, 0.5f, -0.3f, 0.1f, 0.0f, -0.4f], 1, 6); + var before = (float[])net.Forward(x).Data.Clone(); + + var deeper = net.Deepen(new Xoshiro256StarStar(4)); // Net2DeeperNet (identity-init layer) + Assert.Equal([12, 12, 12], deeper.HiddenSizes); + var after = deeper.Forward(x).Data; + for (int a = 0; a < before.Length; a++) Assert.Equal(before[a], after[a], 3); + } + [Fact] public void QNetCheckpoint_RoundTrips_BothNetworkTypes() { diff --git a/tests/MintPlayer.AI.ReinforcementLearning.Tests/PolicyNetGrowthTests.cs b/tests/MintPlayer.AI.ReinforcementLearning.Tests/PolicyNetGrowthTests.cs new file mode 100644 index 0000000..3b7bf54 --- /dev/null +++ b/tests/MintPlayer.AI.ReinforcementLearning.Tests/PolicyNetGrowthTests.cs @@ -0,0 +1,90 @@ +using System.Text; +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; + +namespace MintPlayer.AI.ReinforcementLearning.Tests; + +public class PolicyNetGrowthTests +{ + private static Tensor RandomObs(int size, ulong seed) + { + var rng = new Xoshiro256StarStar(seed); + var data = new float[size]; + for (int i = 0; i < size; i++) data[i] = (float)(rng.NextDouble() * 2 - 1); + return new Tensor(data, 1, size); + } + + [Fact] + public void PolicyValueNet_WidenTo_IsFunctionPreserving() + { + var net = new PolicyValueNet(20, [12, 16], 5, new Xoshiro256StarStar(1)); + var x = RandomObs(20, 42); + var (l0, v0) = net.Forward(x); + var logits0 = (float[])l0.Data.Clone(); float val0 = v0.Data[0]; + + var wider = net.WidenTo([24, 40], new Xoshiro256StarStar(2)); // Net2WiderNet + Assert.Equal([24, 40], wider.Trunk); + var (l1, v1) = wider.Forward(x); + for (int a = 0; a < logits0.Length; a++) Assert.Equal(logits0[a], l1.Data[a], 3); + Assert.Equal(val0, v1.Data[0], 3); + } + + [Fact] + public void PolicyValueNet_Deepen_IsFunctionPreserving() + { + var net = new PolicyValueNet(20, [16, 16], 5, new Xoshiro256StarStar(3)); + var x = RandomObs(20, 7); + var (l0, v0) = net.Forward(x); + var logits0 = (float[])l0.Data.Clone(); float val0 = v0.Data[0]; + + var deeper = net.Deepen(new Xoshiro256StarStar(4)); // Net2DeeperNet (identity-init layer) + Assert.Equal([16, 16, 16], deeper.Trunk); + var (l1, v1) = deeper.Forward(x); + for (int a = 0; a < logits0.Length; a++) Assert.Equal(logits0[a], l1.Data[a], 3); + Assert.Equal(val0, v1.Data[0], 3); + } + + [Fact] + public void CubePolicyNet_LoadsLegacyV1Checkpoint() + { + // A shipped cube.policy.ckpt is v1: header + one hidden width + the four layers' floats (trunk1, trunk2, + // policy head, value head), which is exactly Parameters() order. Hand-write that layout and confirm the + // refactored (variable-depth) loader still reads it and reproduces the forward pass. + var reference = new CubePolicyNet(new Xoshiro256StarStar(9), hidden: 8); + using var ms = new MemoryStream(); + using (var w = new BinaryWriter(ms, Encoding.UTF8, leaveOpen: true)) + { + CheckpointFormat.WriteHeader(w, CubePolicyNet.CheckpointKind, 1); + w.Write(8); // single hidden width (v1) + foreach (var p in reference.Parameters()) CheckpointFormat.WriteFloats(w, p.Data); + } + ms.Position = 0; + + var loaded = CubePolicyNet.Load(ms); + Assert.Equal([8, 8], loaded.Trunk); + var x = RandomObs(RubiksCubeEnv.ObservationSize, 123); + var (lr, vr) = reference.Forward(x); + var (ll, vl) = loaded.Forward(x); + Assert.Equal(lr.Data, ll.Data); + Assert.Equal(vr.Data[0], vl.Data[0]); + } + + [Fact] + public void CubePolicyNet_GrownCheckpoint_RoundTrips() + { + var net = new CubePolicyNet(new Xoshiro256StarStar(5), hidden: 8) + .WidenTo([16, 16], new Xoshiro256StarStar(6)) + .Deepen(new Xoshiro256StarStar(7)); // → [16,16,16] + using var ms = new MemoryStream(); + net.Save(ms); + ms.Position = 0; + var loaded = CubePolicyNet.Load(ms); + + Assert.Equal([16, 16, 16], loaded.Trunk); + var x = RandomObs(RubiksCubeEnv.ObservationSize, 321); + Assert.Equal(net.Forward(x).Logits.Data, loaded.Forward(x).Logits.Data); + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviCampaign.cs index 0411a22..332e820 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviCampaign.cs @@ -3,6 +3,7 @@ using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Planning; using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; using MintPlayer.AI.ReinforcementLearning.Ilgpu; @@ -61,7 +62,7 @@ internal sealed record CubeDaviSettings /// eval-only modes (greedy / A* / BWAS / time-budget / value-curve). The two depth-column CSVs are campaign-owned. /// /// -internal sealed class CubeDaviCampaign(AdaptiveBackend adaptive, CubeDaviSettings settings) : ITrainingCampaign +internal sealed class CubeDaviCampaign(AdaptiveBackend adaptive, CubeDaviSettings settings) : ITrainingCampaign, INetworkTelemetrySource { private const int TrainChunkIterations = 1000; // P.8: train in 1000-iter chunks so the GPU-idle eval runs less private const long ProbeEvery = 15_000; // iters between in-loop BWAS capability probes @@ -98,6 +99,19 @@ internal sealed class CubeDaviCampaign(AdaptiveBackend adaptive, CubeDaviSetting public long Samples => _totalIterations * _s.BatchSize; public bool IsComplete => _s.TargetSamples > 0 && Samples >= _s.TargetSamples; + // --- Live telemetry (INetworkTelemetrySource): read-only; a viewer samples the current net as it trains. + // On the GPU-resident path the CPU net is the periodically-synced master, so this reads the last synced weights. --- + string INetworkTelemetrySource.NetKind => _valueId; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => ReferenceEquals(_net, null) ? null : [.. _net.Parameters()]; + NetworkMetrics INetworkTelemetrySource.Sample() => new(Samples, _s.TargetSamples, _lastLoss, _curriculumDepth, double.NaN); + // Single scalar output = the learned cost-to-go; forward a fixed scramble so its estimate + hidden activations + // are visible live (the GPU-resident weights are mirrored by this CPU master, read lock-free). + IReadOnlyList? INetworkTelemetrySource.OutputLabels => ["Estimated distance to solved (quarter-turn moves)"]; + (float[] Input, float[] Output)? INetworkTelemetrySource.SampleIo() => CubeViz.SampleValueIo(_net, ref _probeObs); + float[][]? INetworkTelemetrySource.SampleActivations() => CubeViz.SampleValueActivations(_net, ref _probeObs); + private float[]? _probeObs; + public bool Resume(IModelStore store) { _store = store; diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviLab.cs index 9d4f434..8a194e4 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeDaviLab.cs @@ -1,4 +1,5 @@ using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Hosting; @@ -165,11 +166,15 @@ public static void Run(string[] args) var store = host.Services.GetRequiredService(); var runner = host.Services.GetRequiredService(); var backend = host.Services.GetRequiredService(); - runner.Run(new CubeDaviCampaign(backend, settings), store, new CampaignOptions + + var campaign = new CubeDaviCampaign(backend, settings); + using var viz = VizLauncher.TryStart(args, campaign, host.Services.GetRequiredService()); + runner.Run(campaign, store, new CampaignOptions { Duration = TimeSpan.FromHours(hours), EvalOnly = evalOnly, OnEval = CampaignCli.Console(), }); + SnakeLab.WaitForViewer(viz); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeEfficientCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeEfficientCampaign.cs index 332e43c..f7fe8d5 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeEfficientCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeEfficientCampaign.cs @@ -3,6 +3,7 @@ using MintPlayer.AI.ReinforcementLearning.Core.Nn; using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; using MintPlayer.AI.ReinforcementLearning.Ilgpu; @@ -17,9 +18,10 @@ /// the Ilgpu.Hosting AddGpuBackend(); the container owns its lifetime). Resumes net + Adam + a persisted /// (samples, round) counter under distinct `policy-efficient*` ids, so the imitation net is never touched. /// -internal sealed class CubeEfficientCampaign(AdaptiveBackend adaptive, ulong seed, float learningRate, int width, int maxScramble, int beamWidth, int evalEpisodes) - : ITrainingCampaign +internal sealed class CubeEfficientCampaign(AdaptiveBackend adaptive, ulong seed, float learningRate, int width, int maxScramble, int beamWidth, int evalEpisodes, bool grow = false, int growEvery = 50_000) + : ITrainingCampaign, INetworkTelemetrySource { + private readonly Xoshiro256StarStar _growRng = new(seed ^ 0x6C0FFEEUL); // dedicated stream for growth private const string PolicyId = "policy-efficient"; private const string PolicyAdamId = "policy-efficient-adam"; private const string PolicyProgressId = "policy-efficient-progress"; @@ -36,6 +38,7 @@ internal sealed class CubeEfficientCampaign(AdaptiveBackend adaptive, ulong seed private long _round, _totalSamples; private double _windowCe, _windowHuber, _windowAcc; private long _windowCount; + private double _liveLoss = double.NaN, _liveAcc = double.NaN; // most-recent batch, for the live viewer public string Environment => CubeIds.Environment; @@ -56,8 +59,11 @@ public bool Resume(IModelStore store) } else { - _net = new CubePolicyNet(new Xoshiro256StarStar(seed ^ 0xC0FFEE), hidden: width); - Log($"initialized a fresh EfficientCube net '{PolicyId}' (trunk width {width})"); + var initRng = new Xoshiro256StarStar(seed ^ 0xC0FFEE); + _net = grow ? new CubePolicyNet(initRng, DqnGrowth.Start) : new CubePolicyNet(initRng, hidden: width); + Log(grow + ? $"initialized a fresh GROWING EfficientCube net '{PolicyId}' (start trunk [{string.Join(",", DqnGrowth.Start)}])" + : $"initialized a fresh EfficientCube net '{PolicyId}' (trunk width {width})"); resumed = false; } } @@ -114,7 +120,11 @@ public long TrainChunk() _windowAcc += acc; _windowCount++; _totalSamples += BatchSize; + _liveLoss = ce + huber; + _liveAcc = acc; } + if (PolicyGrowth.Maybe(_net, _totalSamples, grow, growEvery, learningRate, _growRng, Log) is var g && g.HasValue) + (_net, _adam) = (g.Value.Net, g.Value.Adam); return _totalSamples; } @@ -195,5 +205,15 @@ public void Checkpoint(IModelStore store) // device-resident DeviceMlp is created and disposed inside Evaluate, so there is nothing campaign-owned here. public void Dispose() { } + // --- Live telemetry (INetworkTelemetrySource): read-only; a viewer samples the current net as it trains. --- + string INetworkTelemetrySource.NetKind => "cube-policy"; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => ReferenceEquals(_net, null) ? null : [.. _net.Parameters()]; + NetworkMetrics INetworkTelemetrySource.Sample() => new(_totalSamples, 0, _liveLoss, _liveAcc, double.NaN); + IReadOnlyList? INetworkTelemetrySource.OutputLabels => RubiksCubeEnv.ActionLabels; // 12 quarter-turns + (float[] Input, float[] Output)? INetworkTelemetrySource.SampleIo() => CubeViz.SampleIo(_net, ref _probeObs); + float[][]? INetworkTelemetrySource.SampleActivations() => CubeViz.SampleActivations(_net, ref _probeObs); + private float[]? _probeObs; + private static void Log(string message) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} {message}"); } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeImitationCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeImitationCampaign.cs index 19c496d..8527dfd 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeImitationCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeImitationCampaign.cs @@ -1,7 +1,9 @@ using System.Text; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; @@ -12,8 +14,9 @@ /// quarter-turn + Huber on distance-to-go), and reports per-depth greedy/search solve rates. Resumes the net + /// Adam from `cube.policy` / `cube.policy-adam` (width-ladder ids via ). /// -internal sealed class CubeImitationCampaign(ulong seed, float learningRate, int width) : ITrainingCampaign +internal sealed class CubeImitationCampaign(ulong seed, float learningRate, int width, bool grow = false, int growEvery = 4096) : ITrainingCampaign, INetworkTelemetrySource { + private readonly Xoshiro256StarStar _growRng = new(seed ^ 0x6C0FFEEUL); // dedicated stream for growth private const int BatchSize = 256; private const int SamplesPerRound = 4096; private static readonly int[] EvalDepths = [2, 4, 6, 8, 10, 12, 16, 20]; @@ -27,6 +30,7 @@ internal sealed class CubeImitationCampaign(ulong seed, float learningRate, int private long _round, _totalSamples, _totalSolves; private double _windowCe, _windowHuber, _windowAcc; private long _windowCount; + private double _liveLoss = double.NaN, _liveAcc = double.NaN; // most-recent batch, for the live viewer public string Environment => CubeIds.Environment; @@ -43,8 +47,11 @@ public bool Resume(IModelStore store) } else { - _net = new CubePolicyNet(new Xoshiro256StarStar(seed ^ 0xDEADBEEF), hidden: width); - Log($"initialized a fresh cube policy net '{_ids.Policy}' (trunk width {width})"); + var initRng = new Xoshiro256StarStar(seed ^ 0xDEADBEEF); + _net = grow ? new CubePolicyNet(initRng, DqnGrowth.Start) : new CubePolicyNet(initRng, hidden: width); + Log(grow + ? $"initialized a fresh GROWING cube policy net '{_ids.Policy}' (start trunk [{string.Join(",", DqnGrowth.Start)}])" + : $"initialized a fresh cube policy net '{_ids.Policy}' (trunk width {width})"); resumed = false; } } @@ -97,7 +104,11 @@ public long TrainChunk() _windowAcc += acc; _windowCount++; _totalSamples += BatchSize; + _liveLoss = ce + huber; + _liveAcc = acc; } + if (PolicyGrowth.Maybe(_net, _totalSamples, grow, growEvery, learningRate, _growRng, Log) is var g && g.HasValue) + (_net, _adam) = (g.Value.Net, g.Value.Adam); return _totalSamples; } @@ -192,5 +203,17 @@ private void EvaluateGate() Log($"gate: {totalSolved}/100 solved ({totalSolved}%, target >= 90%); greedy alone {totalGreedy}%"); } + // --- Live telemetry (INetworkTelemetrySource): read-only; a viewer samples the current net as it trains. --- + string INetworkTelemetrySource.NetKind => "cube-policy"; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => ReferenceEquals(_net, null) ? null : [.. _net.Parameters()]; + NetworkMetrics INetworkTelemetrySource.Sample() => new(_totalSamples, 0, _liveLoss, _liveAcc, double.NaN); + IReadOnlyList? INetworkTelemetrySource.OutputLabels => RubiksCubeEnv.ActionLabels; // 12 quarter-turns + // No running env, so the viewer forwards a FIXED scramble each frame — you watch the net's move preferences + + // hidden activations for that one board evolve as it learns. + (float[] Input, float[] Output)? INetworkTelemetrySource.SampleIo() => CubeViz.SampleIo(_net, ref _probeObs); + float[][]? INetworkTelemetrySource.SampleActivations() => CubeViz.SampleActivations(_net, ref _probeObs); + private float[]? _probeObs; + private static void Log(string message) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} {message}"); } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeLab.cs index 57e023b..d8116e1 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeLab.cs @@ -1,5 +1,6 @@ using System.Globalization; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Hosting; @@ -19,6 +20,8 @@ public static void Run(string[] args) float learningRate = 3e-4f; bool evalOnly = false; int width = 512; + bool grow = false; // --grow : progressively grow the net wider+deeper mid-training (Net2Net) + int growEvery = 4096; // --grow-every : samples between growth steps (with --grow) for (int i = 0; i < args.Length; i++) { if (args[i] == "--hours" && i + 1 < args.Length) hours = double.Parse(args[++i], CultureInfo.InvariantCulture); @@ -27,6 +30,8 @@ public static void Run(string[] args) else if (args[i] == "--lr" && i + 1 < args.Length) learningRate = float.Parse(args[++i], CultureInfo.InvariantCulture); else if (args[i] == "--width" && i + 1 < args.Length) width = int.Parse(args[++i]); else if (args[i] == "--eval-only") evalOnly = true; + else if (args[i] == "--grow") grow = true; + else if (args[i] == "--grow-every" && i + 1 < args.Length) growEvery = int.Parse(args[++i]); } // DI all the way: the model store, clock and CampaignRunner are resolved from the AIHost container. @@ -34,12 +39,16 @@ public static void Run(string[] args) var store = host.Services.GetRequiredService(); var runner = host.Services.GetRequiredService(); string csvPath = Path.Combine(dataDir, "logs", "cube-imitation.csv"); - runner.Run(new CubeImitationCampaign(seed, learningRate, width), store, new CampaignOptions + + var campaign = new CubeImitationCampaign(seed, learningRate, width, grow, growEvery); + using var viz = VizLauncher.TryStart(args, campaign, host.Services.GetRequiredService()); + runner.Run(campaign, store, new CampaignOptions { Duration = TimeSpan.FromHours(hours), EvalOnly = evalOnly, OnEval = CampaignCli.ConsoleAndCsv(csvPath), }); + SnakeLab.WaitForViewer(viz); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubePolicyLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubePolicyLab.cs index e60ed87..df399dd 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubePolicyLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubePolicyLab.cs @@ -1,5 +1,6 @@ using System.Globalization; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Hosting; @@ -24,6 +25,8 @@ public static void Run(string[] args) int beamWidth = 2_000; int evalEpisodes = 20; bool evalOnly = false; + bool grow = false; // --grow : progressively grow the net wider+deeper mid-training (Net2Net) + int growEvery = 50_000; // --grow-every : samples between growth steps (with --grow) for (int i = 0; i < args.Length; i++) { if (args[i] == "--hours" && i + 1 < args.Length) hours = double.Parse(args[++i], CultureInfo.InvariantCulture); @@ -35,6 +38,8 @@ public static void Run(string[] args) else if (args[i] == "--beam" && i + 1 < args.Length) beamWidth = int.Parse(args[++i]); else if (args[i] == "--episodes" && i + 1 < args.Length) evalEpisodes = int.Parse(args[++i]); else if (args[i] == "--eval-only") evalOnly = true; + else if (args[i] == "--grow") grow = true; + else if (args[i] == "--grow-every" && i + 1 < args.Length) growEvery = int.Parse(args[++i]); } // DI all the way: the model store, clock, GPU backend and CampaignRunner are resolved from the AIHost @@ -46,14 +51,16 @@ public static void Run(string[] args) var runner = host.Services.GetRequiredService(); var backend = host.Services.GetRequiredService(); string csvPath = Path.Combine(dataDir, "logs", "cube-policy.csv"); - runner.Run( - new CubeEfficientCampaign(backend, seed, learningRate, width, maxScramble, beamWidth, evalEpisodes), - store, + + var campaign = new CubeEfficientCampaign(backend, seed, learningRate, width, maxScramble, beamWidth, evalEpisodes, grow, growEvery); + using var viz = VizLauncher.TryStart(args, campaign, host.Services.GetRequiredService()); + runner.Run(campaign, store, new CampaignOptions { Duration = TimeSpan.FromHours(hours), EvalOnly = evalOnly, OnEval = CampaignCli.ConsoleAndCsv(csvPath), }); + SnakeLab.WaitForViewer(viz); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeViz.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeViz.cs new file mode 100644 index 0000000..50ab993 --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/CubeViz.cs @@ -0,0 +1,67 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; +using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Environments.RubiksCube; + +/// +/// Shared live-viewer helpers for the cube campaigns. They train on shuffled scramble batches, not a running env, +/// so there's no "current observation" — instead the viewer forwards ONE fixed scramble (a depth-8 board from a +/// constant seed) each frame, so a viewer watches the net's move preferences + hidden activations for that single +/// board evolve as it learns. All forwards are read-only (no Backward) and, for a single row, stay on the CPU even +/// under the GPU backend (well below its MAC threshold) — so they never contend with training. +/// +internal static class CubeViz +{ + private static float[] Probe(ref float[]? cache) + { + if (cache is null) + { + var cube = new FaceletCube(); + cube.Apply(FaceletCube.ScrambleMoves(new Xoshiro256StarStar(0xC0FFEE), 8, quarterTurnsOnly: true)); + var obs = new float[RubiksCubeEnv.ObservationSize]; + RubiksCubeEnv.WriteObservation(cube, obs); + cache = obs; + } + return cache; + } + + /// Policy net: the fixed board + its 12 move logits. + public static (float[] Input, float[] Output)? SampleIo(CubePolicyNet? net, ref float[]? cache) + { + if (net is null) return null; + try + { + var obs = Probe(ref cache); + var (logits, _) = net.Forward(new Tensor((float[])obs.Clone(), 1, obs.Length)); + return ((float[])obs.Clone(), [.. logits.Data]); + } + catch { return null; } + } + + public static float[][]? SampleActivations(CubePolicyNet? net, ref float[]? cache) + { + if (net is null) return null; + try { var obs = Probe(ref cache); return net.LayerActivations(new Tensor((float[])obs.Clone(), 1, obs.Length)); } + catch { return null; } + } + + /// Value net (DAVI): the fixed board + its single cost-to-go estimate. + public static (float[] Input, float[] Output)? SampleValueIo(IValueNet? net, ref float[]? cache) + { + if (net is null) return null; + try + { + var obs = Probe(ref cache); + var y = net.Forward(new Tensor((float[])obs.Clone(), 1, obs.Length)); + return ((float[])obs.Clone(), [.. y.Data]); + } + catch { return null; } + } + + public static float[][]? SampleValueActivations(IValueNet? net, ref float[]? cache) + { + if (net is not ResidualMlp res) return null; + try { var obs = Probe(ref cache); return res.LayerActivations(new Tensor((float[])obs.Clone(), 1, obs.Length)); } + catch { return null; } + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/DqnGrowth.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/DqnGrowth.cs new file mode 100644 index 0000000..9e7c98d --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/DqnGrowth.cs @@ -0,0 +1,51 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +/// +/// Progressive architecture growth for any DQN campaign (a in a +/// ): on a step cadence it grows the net toward larger stages, each a single +/// function-preserving Net2WiderNet (wider) or Net2DeeperNet (one more layer) step — so capacity is added mid-run +/// with no loss spike. Shared by every DQN game (Snake, FruitCake). The schedule starts tiny and alternates +/// wider → deeper so a viewer literally watches the graph grow both ways. +/// +internal static class DqnGrowth +{ + /// Each stage is exactly one widen (same depth, larger widths) or deepen (one extra layer) from the + /// previous. is the architecture a growing run should be constructed with. + public static readonly int[][] Stages = [[16], [32], [32, 32], [64, 64], [64, 64, 64], [128, 128, 128]]; + public static int[] Start => Stages[0]; + + /// Grows 's net toward the stage its step count has reached (a no-op unless + /// and the net is a non-noisy ). Returns the (possibly new) + /// state; the caller reassigns its _state. + public static DqnTrainingState Maybe(DqnTrainingState state, bool grow, int growEvery, float learningRate, + Xoshiro256StarStar rng, Action log) + { + if (!grow || state.Online is not DuelingQNet online || online.Noisy) return state; + int target = Math.Min(Stages.Length - 1, state.StepsCompleted / Math.Max(1, growEvery)); + int stage = CurrentStage(online.Trunk); + while (stage < target) { state = GrowTo(state, Stages[stage + 1], learningRate, rng, log); stage++; } + return state; + } + + private static int CurrentStage(int[] hidden) + { + for (int s = Stages.Length - 1; s >= 0; s--) + if (Stages[s].AsSpan().SequenceEqual(hidden)) return s; + return 0; // an architecture not on the schedule (e.g. a non-grow resume) — treat as the start + } + + private static DqnTrainingState GrowTo(DqnTrainingState state, int[] next, float learningRate, + Xoshiro256StarStar rng, Action log) + { + var online = (DuelingQNet)state.Online; + bool deeper = next.Length > online.Trunk.Length; + var grownOnline = deeper ? online.Deepen(rng) : online.WidenTo(next, rng); + var grownTarget = (DuelingQNet)grownOnline.CloneStructure(); + grownTarget.CopyFrom(grownOnline); // re-sync the target net on an architecture change + var grown = state.WithNetwork(grownOnline, grownTarget, new Adam(grownOnline.Parameters(), learningRate)); + log($"{(deeper ? "deepened" : "widened")} net → [{string.Join(",", grownOnline.HiddenSizes)}] at {grown.StepsCompleted:N0} steps (function-preserving)"); + return grown; + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeDqnCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeDqnCampaign.cs index 48ca4a2..9a7d967 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeDqnCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeDqnCampaign.cs @@ -1,7 +1,9 @@ using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Random; using MintPlayer.AI.ReinforcementLearning.Core.Schedules; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.FruitCake; @@ -14,9 +16,11 @@ /// and continues. Persists the deployable net under `fruitcake`/`dqn` (the id /// the web's FruitCakeModelService will load, A3) plus the full resume state under `fruitcake`/`dqn-state`. /// -internal sealed class FruitCakeDqnCampaign(ulong seed, int chunkSteps, long targetSteps, int evalEpisodes, float learningRate, float epsilonStart, int[] hidden, double gamma, bool noisy = false, int nStep = 1, bool shapeRewards = false) - : ITrainingCampaign +internal sealed class FruitCakeDqnCampaign(ulong seed, int chunkSteps, long targetSteps, int evalEpisodes, float learningRate, float epsilonStart, int[] hidden, double gamma, bool noisy = false, int nStep = 1, bool shapeRewards = false, bool grow = false, int growEvery = 2000) + : ITrainingCampaign, INetworkTelemetrySource { + // Progressive-growth demo (shared with Snake via DqnGrowth): grow the net wider+deeper on the growEvery cadence. + private readonly Xoshiro256StarStar _growRng = new(seed ^ 0x9E3779B97F4A7C15UL); private const string NetId = "dqn"; // deployable DuelingQNet — the id the web loads (shared by both lines) // Noisy training is a SEPARATE line with its own resume state: it must never resume a PLAIN state as a plain // net (which would then train with ε=0 and NO exploration). The deployable NetId is shared and keep-best gated. @@ -120,6 +124,7 @@ public long TrainChunk() var options = BaseOptions with { MaxSteps = to, EvalEvery = Math.Max(1, chunkSteps) }; var result = DqnTrainer.Train(_env, options, _seeds, resume: _state, warmStart: _state is null ? _warmNet : null); _state = result.State; + _state = DqnGrowth.Maybe(_state, grow, growEvery, learningRate, _growRng, Log); return _state.StepsCompleted; } @@ -192,5 +197,40 @@ public void Dispose() { } return (totalScore / evalEpisodes, totalMaxTier / evalEpisodes, totalReturn / evalEpisodes); } + // --- Live telemetry (INetworkTelemetrySource): read-only; a viewer samples the current net as it trains. --- + string INetworkTelemetrySource.NetKind => "dueling-q"; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => (_state?.Online ?? _warmNet) is { } net ? [.. net.Parameters()] : null; + NetworkMetrics INetworkTelemetrySource.Sample() + => new(_state?.StepsCompleted ?? 0, targetSteps, _state?.LastLoss ?? double.NaN, _lastEvalScore, double.NaN); + + // Environment-aware neuron labels + live values, so the viewer can say what each input/output MEANS and its + // current value: each input is a named observation feature; each output is the Q-value of dropping in a column. + IReadOnlyList? INetworkTelemetrySource.InputLabels => FruitCakeEnv.ObservationLabels; + IReadOnlyList? INetworkTelemetrySource.OutputLabels => FruitCakeEnv.ActionLabels; + (float[] Input, float[] Output)? INetworkTelemetrySource.SampleIo() + { + var net = _state?.Online ?? _warmNet; + var obs = _state?.CurrentObs; + if (net is null || obs is null || obs.Length != FruitCakeEnv.ObservationSize) return null; + try + { + // Forward the most-recent observation to get the net's current per-column Q-values. Read-only: it never + // writes weights/grads (no Backward), so a concurrent training step is unaffected (torn reads only + // flicker a displayed number). A fresh input array so the viewer never sees it mutated mid-serialize. + var input = (float[])obs.Clone(); + float[] q = net.Forward(new Tensor(input, 1, input.Length)).Data; + return (input, q.AsSpan().ToArray()); + } + catch { return null; } + } + float[][]? INetworkTelemetrySource.SampleActivations() + { + var obs = _state?.CurrentObs; + if ((_state?.Online ?? _warmNet) is not DuelingQNet dqn || obs is null || obs.Length != FruitCakeEnv.ObservationSize) return null; + try { return dqn.LayerActivations(new Tensor((float[])obs.Clone(), 1, obs.Length)); } + catch { return null; } + } + private static void Log(string message) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} {message}"); } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeLab.cs index 14a63e3..c5417a1 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/FruitCakeLab.cs @@ -1,5 +1,6 @@ using System.Globalization; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Hosting; @@ -36,6 +37,8 @@ public static void Run(string[] args) int topK = 5; // --topk : depth-2 first-ply expansion width int topK2 = 3; // --topk2 : deeper-ply expansion width (depth 3) string leaf = "net"; // --leaf : search leaf value (net | height | tierpot | blend) + bool grow = false; // --grow : progressively grow the net wider+deeper mid-training (Net2Net demo) + int growEvery = 2000; // --grow-every : drops between growth steps (with --grow) for (int i = 0; i < args.Length; i++) { if (args[i] == "--hours" && i + 1 < args.Length) hours = double.Parse(args[++i], CultureInfo.InvariantCulture); @@ -60,8 +63,13 @@ public static void Run(string[] args) else if (args[i] == "--topk" && i + 1 < args.Length) topK = int.Parse(args[++i]); else if (args[i] == "--topk2" && i + 1 < args.Length) topK2 = int.Parse(args[++i]); else if (args[i] == "--leaf" && i + 1 < args.Length) leaf = args[++i]; + else if (args[i] == "--grow") grow = true; + else if (args[i] == "--grow-every" && i + 1 < args.Length) growEvery = int.Parse(args[++i]); } + // A growing run starts from the tiny first stage and adds capacity mid-training (Net2Wider/DeeperNet). + if (grow) hidden = DqnGrowth.Start; + if (searchEval) { // F1: does forward-model search beat the plain net on max-tier? No training/host needed. @@ -80,14 +88,16 @@ public static void Run(string[] args) var store = host.Services.GetRequiredService(); var runner = host.Services.GetRequiredService(); string csvPath = Path.Combine(dataDir, "logs", "fruitcake-dqn.csv"); - runner.Run( - new FruitCakeDqnCampaign(seed, chunkSteps, targetSteps, evalEpisodes, learningRate, explore, hidden, gamma, noisy, nStep, shape), - store, + + var campaign = new FruitCakeDqnCampaign(seed, chunkSteps, targetSteps, evalEpisodes, learningRate, explore, hidden, gamma, noisy, nStep, shape, grow, growEvery); + using var viz = VizLauncher.TryStart(args, campaign, host.Services.GetRequiredService()); + runner.Run(campaign, store, new CampaignOptions { Duration = TimeSpan.FromHours(hours), EvalOnly = evalOnly, OnEval = CampaignCli.ConsoleAndCsv(csvPath), }); + SnakeLab.WaitForViewer(viz); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyGrowth.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyGrowth.cs new file mode 100644 index 0000000..fca8bef --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/PolicyGrowth.cs @@ -0,0 +1,39 @@ +using MintPlayer.AI.ReinforcementLearning.Core.Nn; +using MintPlayer.AI.ReinforcementLearning.Core.Random; + +/// +/// Progressive architecture growth for the imitation policy campaigns (a two-headed +/// trained with a plain ): on a sample cadence it grows the net toward larger stages, each a single +/// function-preserving Net2WiderNet / Net2DeeperNet step. Shares the schedule with the +/// DQN games. Returns the grown net + a fresh optimizer (Adam moments are keyed to the parameter set) when it grows, +/// else null — the caller reassigns its net/optimizer. +/// +internal static class PolicyGrowth +{ + public static (TNet Net, Adam Adam)? Maybe(TNet net, long samples, bool grow, int growEvery, + float learningRate, Xoshiro256StarStar rng, Action log) + where TNet : IGrowableTrunkNet + { + if (!grow) return null; + int target = Math.Min(DqnGrowth.Stages.Length - 1, (int)(samples / Math.Max(1, growEvery))); + int stage = CurrentStage(net.Trunk); + if (stage >= target) return null; + + var grown = net; + while (stage < target) + { + var next = DqnGrowth.Stages[stage + 1]; + grown = next.Length > grown.Trunk.Length ? grown.Deepen(rng) : grown.WidenTo(next, rng); + stage++; + } + log($"grew policy net → [{string.Join(",", grown.Trunk)}] at {samples:N0} samples (function-preserving)"); + return (grown, new Adam(grown.Parameters(), learningRate)); + } + + private static int CurrentStage(int[] hidden) + { + for (int s = DqnGrowth.Stages.Length - 1; s >= 0; s--) + if (DqnGrowth.Stages[s].AsSpan().SequenceEqual(hidden)) return s; + return 0; // an architecture not on the schedule (e.g. a non-grow resume) — treat as the start + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs index 68d16e5..ac77576 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/Program.cs @@ -3,9 +3,17 @@ // the runner owns the loop/resume/eval cadence/checkpointing, the per-game *Lab files own the flags. // // Usage: MintPlayer.AI.ReinforcementLearning.Lab [--game rushhour|snake|fruitcake|cube|cube-policy|cube-davi] -// [--hours H] [--data DIR] [--seed S] [--lr LR] [--eval-only] ... +// [--hours H] [--data DIR] [--seed S] [--lr LR] [--eval-only] +// [--viz [port]] ... // Default game: rushhour (the original Kociemba-free BFS-oracle imitation campaign, PLAN M16). +// The Lab is a development tool: default to the Development environment (unless the operator set one) so the +// `--viz` live network viewer — gated to Development on purpose — works out of the box. Set +// DOTNET_ENVIRONMENT=Production to run without ever exposing the viewer socket. +if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable("DOTNET_ENVIRONMENT")) + && string.IsNullOrEmpty(Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT"))) + Environment.SetEnvironmentVariable("DOTNET_ENVIRONMENT", "Development"); + for (int i = 0; i < args.Length; i++) { if (args[i] == "--game" && i + 1 < args.Length) diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourImitationCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourImitationCampaign.cs index 5a8d580..25c3d22 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourImitationCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourImitationCampaign.cs @@ -2,6 +2,7 @@ using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Nn; using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.RushHour; using Tensor = MintPlayer.AI.ReinforcementLearning.Core.Numerics.Tensor; @@ -14,8 +15,9 @@ /// distance) on a DAgger mix of on-policy and stratified samples. Eval tracks the held-out official ThinkFun cards /// (1, 38, 39, 40) with reactive play and policy-guided A*, plus a 30-puzzle random hold-out set. /// -internal sealed class RushHourImitationCampaign(ulong seed, float learningRate) : ITrainingCampaign +internal sealed class RushHourImitationCampaign(ulong seed, float learningRate, bool grow = false, int growEvery = 2048) : ITrainingCampaign, INetworkTelemetrySource { + private readonly Xoshiro256StarStar _growRng = new(seed ^ 0x6C0FFEEUL); // dedicated stream for growth private const int BatchSize = 256; private const int SamplesPerConfig = 1024; private const int MaxStatesPerConfig = 150_000; @@ -29,6 +31,7 @@ internal sealed class RushHourImitationCampaign(ulong seed, float learningRate) private double _windowCe, _windowHuber, _windowAcc; private long _windowCount; private long _windowOnPolicy, _windowDrawn; + private double _liveLoss = double.NaN, _liveAcc = double.NaN; // most-recent batch, for the live viewer // Held-out official ThinkFun cards (never produced by the random generator). private readonly (string Name, RushHourPuzzle Puzzle, int Optimal)[] _cards = @@ -72,8 +75,10 @@ public bool Resume(IModelStore store) } else { - _net = new RushHourPolicyNet(new Xoshiro256StarStar(seed ^ 0xDEADBEEF)); - Log("initialized a fresh policy net"); + var initRng = new Xoshiro256StarStar(seed ^ 0xDEADBEEF); + _net = grow ? new RushHourPolicyNet(initRng, DqnGrowth.Start) : new RushHourPolicyNet(initRng); + Log(grow ? $"initialized a fresh GROWING policy net (start trunk [{string.Join(",", DqnGrowth.Start)}])" + : "initialized a fresh policy net"); resumed = false; } } @@ -118,7 +123,11 @@ public long TrainChunk() _windowAcc += acc; _windowCount++; _totalSamples += BatchSize; + _liveLoss = ce + huber; + _liveAcc = acc; } + if (PolicyGrowth.Maybe(_net, _totalSamples, grow, growEvery, learningRate, _growRng, Log) is var g && g.HasValue) + (_net, _adam) = (g.Value.Net, g.Value.Adam); return _totalSamples; } @@ -310,5 +319,38 @@ private static void Shuffle(IList list, Xoshiro256StarStar rng) private static void Log(string message) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} {message}"); + // --- Live telemetry (INetworkTelemetrySource): read-only; a viewer samples the current net as it trains. --- + string INetworkTelemetrySource.NetKind => "rushhour-policy"; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => ReferenceEquals(_net, null) ? null : [.. _net.Parameters()]; + NetworkMetrics INetworkTelemetrySource.Sample() => new(_totalSamples, 0, _liveLoss, _liveAcc, double.NaN); + IReadOnlyList? INetworkTelemetrySource.OutputLabels => RushHourBoard.ActionLabels; // 32 vehicle×dir moves + // No running env, so the viewer forwards ONE fixed puzzle (the level-1 card's start) each frame — you watch the + // net's move preferences + hidden activations for that board evolve. Read-only forward; CPU (imitation has no GPU). + private float[]? _probeObs; + private float[] ProbeObs() + { + if (_probeObs is null) + { + var puzzle = _cards[0].Puzzle; + var obs = new float[RushHourBoard.ObservationSize]; + RushHourBoard.WriteObservation(puzzle, RushHourBoard.InitialPositions(puzzle), obs); + _probeObs = obs; + } + return _probeObs; + } + (float[] Input, float[] Output)? INetworkTelemetrySource.SampleIo() + { + if (ReferenceEquals(_net, null)) return null; + try { var obs = ProbeObs(); var (logits, _) = _net.Forward(new Tensor((float[])obs.Clone(), 1, obs.Length)); return ((float[])obs.Clone(), [.. logits.Data]); } + catch { return null; } + } + float[][]? INetworkTelemetrySource.SampleActivations() + { + if (ReferenceEquals(_net, null)) return null; + try { var obs = ProbeObs(); return _net.LayerActivations(new Tensor((float[])obs.Clone(), 1, obs.Length)); } + catch { return null; } + } + private sealed record Sample(float[] Obs, float[] MaskOffsets, uint LabelMask, float Distance); } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourLab.cs index 753bd11..b2a0163 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/RushHourLab.cs @@ -1,5 +1,6 @@ using System.Globalization; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Hosting; @@ -18,6 +19,8 @@ public static void Run(string[] args) ulong seed = 1; float learningRate = 3e-4f; bool evalOnly = false; + bool grow = false; // --grow : progressively grow the net wider+deeper mid-training (Net2Net) + int growEvery = 2048; // --grow-every : samples between growth steps (with --grow) for (int i = 0; i < args.Length; i++) { if (args[i] == "--hours" && i + 1 < args.Length) hours = double.Parse(args[++i], CultureInfo.InvariantCulture); @@ -25,6 +28,8 @@ public static void Run(string[] args) else if (args[i] == "--seed" && i + 1 < args.Length) seed = ulong.Parse(args[++i]); else if (args[i] == "--lr" && i + 1 < args.Length) learningRate = float.Parse(args[++i], CultureInfo.InvariantCulture); else if (args[i] == "--eval-only") evalOnly = true; + else if (args[i] == "--grow") grow = true; + else if (args[i] == "--grow-every" && i + 1 < args.Length) growEvery = int.Parse(args[++i]); } // DI all the way: the model store, clock and CampaignRunner are resolved from the AIHost container. @@ -32,11 +37,15 @@ public static void Run(string[] args) var store = host.Services.GetRequiredService(); var runner = host.Services.GetRequiredService(); string csvPath = Path.Combine(dataDir, "logs", "imitation.csv"); - runner.Run(new RushHourImitationCampaign(seed, learningRate), store, new CampaignOptions + + var campaign = new RushHourImitationCampaign(seed, learningRate, grow, growEvery); + using var viz = VizLauncher.TryStart(args, campaign, host.Services.GetRequiredService()); + runner.Run(campaign, store, new CampaignOptions { Duration = TimeSpan.FromHours(hours), EvalOnly = evalOnly, OnEval = CampaignCli.ConsoleAndCsv(csvPath), }); + SnakeLab.WaitForViewer(viz); } } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeDqnCampaign.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeDqnCampaign.cs index 886ec85..9c81395 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeDqnCampaign.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeDqnCampaign.cs @@ -1,7 +1,9 @@ using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Nn; using MintPlayer.AI.ReinforcementLearning.Core.Random; +using MintPlayer.AI.ReinforcementLearning.Core.Numerics; using MintPlayer.AI.ReinforcementLearning.Core.Schedules; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.Snake; @@ -15,9 +17,11 @@ /// net under `snake`/`dqn` (the id the web's SnakeModelService loads) plus the full resume state under /// `snake`/`dqn-state`. /// -internal sealed class SnakeDqnCampaign(ulong seed, int trainGrid, int evalGrid, int chunkSteps, long targetSteps, int evalEpisodes, float learningRate, float epsilonStart, int[] hidden, double gamma, float stepPenalty, bool safeMask) - : ITrainingCampaign +internal sealed class SnakeDqnCampaign(ulong seed, int trainGrid, int evalGrid, int chunkSteps, long targetSteps, int evalEpisodes, float learningRate, float epsilonStart, int[] hidden, double gamma, float stepPenalty, bool safeMask, bool grow = false, int growEvery = 5000) + : ITrainingCampaign, INetworkTelemetrySource { + // Progressive-growth demo (shared with FruitCake via DqnGrowth): grow the net wider+deeper on the growEvery cadence. + private readonly Xoshiro256StarStar _growRng = new(seed ^ 0x9E3779B97F4A7C15UL); private const string NetId = "dqn"; // deployable DuelingQNet — the id the web loads private const string StateId = "dqn-state"; // full DqnTrainingState for lossless resume @@ -104,6 +108,7 @@ public long TrainChunk() // First chunk: resume the full state if present, else warm-start from the deployable net (if any). var result = DqnTrainer.Train(_env, options, _seeds, resume: _state, warmStart: _state is null ? _warmNet : null); _state = result.State; + _state = DqnGrowth.Maybe(_state, grow, growEvery, learningRate, _growRng, Log); return _state.StepsCompleted; } @@ -174,5 +179,35 @@ public void Dispose() { } return (totalFood / evalEpisodes, totalReturn / evalEpisodes); } + // --- Live telemetry (INetworkTelemetrySource): read-only; a viewer samples the current net as it trains. --- + string INetworkTelemetrySource.NetKind => "dueling-q"; + IReadOnlyList? INetworkTelemetrySource.SnapshotParameters() + => (_state?.Online ?? _warmNet) is { } net ? [.. net.Parameters()] : null; + NetworkMetrics INetworkTelemetrySource.Sample() + => new(_state?.StepsCompleted ?? 0, targetSteps, _state?.LastLoss ?? double.NaN, _lastEvalFood, double.NaN); + + // Environment-aware neuron labels + live values: inputs are named observation features (177-dim egocentric + // vision + scalars), outputs are the 4 move directions with their Q-values, hidden neurons show activations. + IReadOnlyList? INetworkTelemetrySource.InputLabels => SnakeEnv.ObservationLabels; + IReadOnlyList? INetworkTelemetrySource.OutputLabels => SnakeEnv.ActionLabels; + (float[] Input, float[] Output)? INetworkTelemetrySource.SampleIo() + { + var obs = _state?.CurrentObs; + if ((_state?.Online ?? _warmNet) is not { } net || obs is null || obs.Length != SnakeEnv.ObservationSize) return null; + try + { + var input = (float[])obs.Clone(); + return (input, net.Forward(new Tensor(input, 1, input.Length)).Data.AsSpan().ToArray()); + } + catch { return null; } + } + float[][]? INetworkTelemetrySource.SampleActivations() + { + var obs = _state?.CurrentObs; + if ((_state?.Online ?? _warmNet) is not DuelingQNet dqn || obs is null || obs.Length != SnakeEnv.ObservationSize) return null; + try { return dqn.LayerActivations(new Tensor((float[])obs.Clone(), 1, obs.Length)); } + catch { return null; } + } + private static void Log(string message) => Console.WriteLine($"{DateTime.UtcNow:HH:mm:ss} {message}"); } diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs index 7be234e..1d9c909 100644 --- a/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/SnakeLab.cs @@ -1,6 +1,7 @@ using System.Diagnostics; using System.Globalization; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; using MintPlayer.AI.ReinforcementLearning.Core.Checkpoints; using MintPlayer.AI.ReinforcementLearning.Core.Training; using MintPlayer.AI.ReinforcementLearning.Environments.Snake; @@ -31,6 +32,8 @@ public static void Run(string[] args) float stepPenalty = -0.01f; // --step-penalty : per-step reward; ~0 removes the efficiency pressure that encourages safe starvation bool safeMask = false; // --safe-mask : forbid moves that flood-fill into a region too small for the body (anti-self-trap shield) bool evalOnly = false; + bool grow = false; // --grow : progressively grow the net wider+deeper mid-training (Net2Net demo) + int growEvery = 5000; // --grow-every : steps between growth steps (with --grow) // --search : skip training and evaluate the net-guided look-ahead planner (M34) instead of greedy Q. The // net is only a leaf tiebreak, so the config defaults reproduce PR #11's shipped depth-20/beam-32 sweep. @@ -68,8 +71,13 @@ public static void Run(string[] args) else if (args[i] == "--step-penalty" && i + 1 < args.Length) stepPenalty = float.Parse(args[++i], CultureInfo.InvariantCulture); else if (args[i] == "--safe-mask") safeMask = true; else if (args[i] == "--eval-only") evalOnly = true; + else if (args[i] == "--grow") grow = true; + else if (args[i] == "--grow-every" && i + 1 < args.Length) growEvery = int.Parse(args[++i]); } + // A growing run starts from the tiny first stage and adds capacity mid-training (Net2Wider/DeeperNet). + if (grow) hidden = DqnGrowth.Start; + if (search) { RunSearchEval(netPath, evalGrid, evalEpisodes, seed, cfg); @@ -81,15 +89,25 @@ public static void Run(string[] args) var store = host.Services.GetRequiredService(); var runner = host.Services.GetRequiredService(); string csvPath = Path.Combine(dataDir, "logs", "snake-dqn.csv"); - runner.Run( - new SnakeDqnCampaign(seed, trainGrid, evalGrid, chunkSteps, targetSteps, evalEpisodes, learningRate, explore, hidden, gamma, stepPenalty, safeMask), - store, + + var campaign = new SnakeDqnCampaign(seed, trainGrid, evalGrid, chunkSteps, targetSteps, evalEpisodes, learningRate, explore, hidden, gamma, stepPenalty, safeMask, grow, growEvery); + using var viz = VizLauncher.TryStart(args, campaign, host.Services.GetRequiredService()); + runner.Run(campaign, store, new CampaignOptions { Duration = TimeSpan.FromHours(hours), EvalOnly = evalOnly, OnEval = CampaignCli.ConsoleAndCsv(csvPath), }); + WaitForViewer(viz); + } + + /// Keep the process (and its live viewer) alive after training so the final net can be inspected. + internal static void WaitForViewer(VizServer? viz) + { + if (viz is null || Console.IsInputRedirected) return; + Console.WriteLine($"training finished — viewer still live at {viz.Url}; press Enter to exit."); + Console.ReadLine(); } /// diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/VizLauncher.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/VizLauncher.cs new file mode 100644 index 0000000..c5847bf --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/VizLauncher.cs @@ -0,0 +1,48 @@ +using Microsoft.Extensions.Hosting; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; +using MintPlayer.AI.ReinforcementLearning.Core.Training; + +/// +/// Shared --viz [port] handling for every game Lab: parses the flag, enforces the safety gate, and — if the +/// campaign can report its network — starts the live viewer. Bare --viz uses port +/// 5250. Kept in one place so all games get the identical behaviour and message. +/// +/// The live socket is restricted to a Development host environment: training is a dev-only activity, so the +/// viewer never comes up in a Production-configured process even if --viz is passed. (The Lab defaults to +/// Development — see Program.cs — so it works out of the box; set DOTNET_ENVIRONMENT=Production to +/// disable it.) A campaign that exposes no is skipped with a note. +/// +/// +internal static class VizLauncher +{ + public static VizServer? TryStart(string[] args, ITrainingCampaign campaign, IHostEnvironment environment) + { + int port = ParsePort(args); + if (port == 0) return null; // --viz not passed + + if (!environment.IsDevelopment()) + { + Console.WriteLine($"[viz] --viz ignored: the live network viewer only runs in a Development environment " + + $"(current: {environment.EnvironmentName}). Unset DOTNET_ENVIRONMENT to enable it."); + return null; + } + if (campaign is not INetworkTelemetrySource source) + { + Console.WriteLine($"[viz] --viz ignored: {campaign.GetType().Name} does not expose network telemetry yet."); + return null; + } + + var server = VizServer.Start(port, source); + Console.WriteLine($"live network viewer: {server.Url} (open it to watch the net evolve)"); + return server; + } + + /// Returns the requested viewer port, 0 when --viz is absent. Bare --viz → 5250. + private static int ParsePort(string[] args) + { + for (int i = 0; i < args.Length; i++) + if (args[i] == "--viz") + return i + 1 < args.Length && int.TryParse(args[i + 1], out int p) ? p : 5250; + return 0; + } +} diff --git a/tools/MintPlayer.AI.ReinforcementLearning.Lab/VizServer.cs b/tools/MintPlayer.AI.ReinforcementLearning.Lab/VizServer.cs new file mode 100644 index 0000000..4dccb5a --- /dev/null +++ b/tools/MintPlayer.AI.ReinforcementLearning.Lab/VizServer.cs @@ -0,0 +1,479 @@ +using System.Net; +using System.Net.WebSockets; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using System.Threading; +using System.Threading.Channels; +using System.Threading.Tasks; +using MintPlayer.AI.ReinforcementLearning.Core.Telemetry; + +/// +/// A self-contained, dependency-free live-network viewer hosted by the training process itself: a tiny +/// on localhost that serves one HTML page (GET /) and streams telemetry to it +/// over a WebSocket (GET /ws). It samples an (the running +/// campaign) on a fixed cadence — a pull model, so it works for every trainer without any of them knowing it +/// exists. A WebSocket (rather than one-way SSE) keeps the channel bidirectional, so the viewer can later send +/// control messages back (pause/step, change cadence, pick a layer) without changing the transport. +/// +/// Fully async: each viewer has its own bounded outbound queue drained by an async send pump, and a background +/// sample loop awaits the cadence timer — there is no blocking network I/O on any hot path. Fault-tolerant: a +/// viewer that never connects, disconnects, or falls behind is dropped (its bounded queue discards stale frames), +/// so telemetry never blocks or breaks training. Development-only — never wired into the deployed web app. +/// +/// +internal sealed class VizServer : IDisposable +{ + private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web) + { + // Metrics can be non-finite (e.g. ε = NaN for a supervised campaign, eval = -Infinity before the first + // eval); emit them as the named JSON literals rather than throwing — the viewer renders non-finite as "—". + NumberHandling = JsonNumberHandling.AllowNamedFloatingPointLiterals, + }; + + /// A connected viewer: its socket plus a bounded outbound queue drained by one async send pump. + /// The queue drops the oldest frame under backpressure, so a slow viewer can never stall the sampler. + private sealed class Client(WebSocket socket) + { + public WebSocket Socket { get; } = socket; + public Channel Outbox { get; } = Channel.CreateBounded( + new BoundedChannelOptions(32) { FullMode = BoundedChannelFullMode.DropOldest, SingleReader = true }); + } + + private readonly HttpListener _listener = new(); + private readonly List _clients = []; + private readonly Lock _gate = new(); + private readonly CancellationTokenSource _shutdown = new(); + private readonly INetworkTelemetrySource _source; + private readonly int _intervalMs; + private volatile bool _running = true; + + public string Url { get; } + + private VizServer(int port, INetworkTelemetrySource source, int intervalMs) + { + _source = source; + _intervalMs = intervalMs; + Url = $"http://localhost:{port}/"; + _listener.Prefixes.Add(Url); + } + + /// Starts the viewer on and begins sampling . + public static VizServer Start(int port, INetworkTelemetrySource source, int intervalMs = 150) + { + var server = new VizServer(port, source, intervalMs); + server._listener.Start(); + _ = Task.Run(server.AcceptLoop); + _ = Task.Run(server.SampleLoop); + return server; + } + + private async Task AcceptLoop() + { + while (_running) + { + HttpListenerContext ctx; + try { ctx = await _listener.GetContextAsync(); } + catch { return; } // listener stopped + _ = Task.Run(() => Handle(ctx)); + } + } + + private async Task Handle(HttpListenerContext ctx) + { + string path = ctx.Request.Url?.AbsolutePath ?? "/"; + if (path == "/ws" && ctx.Request.IsWebSocketRequest) { await ServeWebSocket(ctx); return; } + + var html = Encoding.UTF8.GetBytes(Page); + ctx.Response.ContentType = "text/html; charset=utf-8"; + ctx.Response.ContentLength64 = html.Length; + try { await ctx.Response.OutputStream.WriteAsync(html); } catch { /* client went away */ } + finally { ctx.Response.Close(); } + } + + private async Task ServeWebSocket(HttpListenerContext ctx) + { + WebSocket socket; + try { socket = (await ctx.AcceptWebSocketAsync(subProtocol: null)).WebSocket; } + catch { ctx.Response.Abort(); return; } + + var client = new Client(socket); + lock (_gate) _clients.Add(client); + + // Give the newcomer the current graph at once (before the next sample tick), so it never sits blank. + if (CurrentTopology() is { } topo) client.Outbox.Writer.TryWrite(topo); + + var pump = Task.Run(() => SendPump(client)); + + // Hold the connection open and drain anything the viewer sends (control messages will arrive here later); + // this loop also detects the close so the client can be dropped. + var buffer = new byte[4096]; + try + { + while (socket.State == WebSocketState.Open && !_shutdown.IsCancellationRequested) + { + var result = await socket.ReceiveAsync(buffer, _shutdown.Token); + if (result.MessageType == WebSocketMessageType.Close) break; + } + } + catch { /* client dropped or shutting down */ } + finally { Drop(client); await pump; } + } + + /// Drains one client's outbound queue, awaiting each send — the only place a socket is written. + private async Task SendPump(Client client) + { + try + { + await foreach (var message in client.Outbox.Reader.ReadAllAsync(_shutdown.Token)) + await client.Socket.SendAsync(new ArraySegment(message), WebSocketMessageType.Text, endOfMessage: true, _shutdown.Token); + } + catch { /* client gone or shutting down */ } + finally { Drop(client); } + } + + /// Samples the source on the cadence and broadcasts topology (when it changes) + a weight frame. + private async Task SampleLoop() + { + string? lastTopologyJson = null; + while (!_shutdown.IsCancellationRequested) + { + try { await Task.Delay(_intervalMs, _shutdown.Token); } + catch { break; } + + // Nothing to do (and nothing to pay) while no one is watching. + lock (_gate) { if (_clients.Count == 0) continue; } + + try + { + var parameters = _source.SnapshotParameters(); + if (parameters is null || parameters.Count == 0) continue; + + var topology = NetworkInspector.Describe(parameters, _source.NetKind, _source.InputLabels, _source.OutputLabels); + string topologyJson = JsonSerializer.Serialize(topology, Json); + if (topologyJson != lastTopologyJson) + { + lastTopologyJson = topologyJson; + Broadcast(Envelope("topology", topologyJson)); + } + + var io = _source.SampleIo(); + var frame = NetworkInspector.CaptureFrame(parameters, _source.Sample(), io?.Input, io?.Output, _source.SampleActivations()); + Broadcast(Envelope("frame", JsonSerializer.Serialize(frame, Json))); + } + catch { /* transient (e.g. net swapped mid-sample) — skip this frame */ } + } + } + + /// The current graph as a ready-to-send topology envelope, or null if the net doesn't exist yet. + private byte[]? CurrentTopology() + { + try + { + var parameters = _source.SnapshotParameters(); + if (parameters is null || parameters.Count == 0) return null; + var topology = NetworkInspector.Describe(parameters, _source.NetKind, _source.InputLabels, _source.OutputLabels); + return Envelope("topology", JsonSerializer.Serialize(topology, Json)); + } + catch { return null; } + } + + private void Broadcast(byte[] message) + { + Client[] snapshot; + lock (_gate) snapshot = [.. _clients]; + foreach (var c in snapshot) c.Outbox.Writer.TryWrite(message); // non-blocking; DropOldest handles backpressure + } + + // WebSocket has no SSE-style event names, so each message self-describes: {"type":,"data":}. + // `json` is already-serialized, so this splices it in without a second serialize pass. + private static byte[] Envelope(string type, string json) + => Encoding.UTF8.GetBytes($"{{\"type\":\"{type}\",\"data\":{json}}}"); + + private void Drop(Client client) + { + lock (_gate) { if (!_clients.Remove(client)) return; } + client.Outbox.Writer.TryComplete(); // ends the send pump's ReadAllAsync + try { client.Socket.Abort(); } catch { } + try { client.Socket.Dispose(); } catch { } + } + + public void Dispose() + { + _running = false; + _shutdown.Cancel(); + lock (_gate) + { + foreach (var c in _clients) { c.Outbox.Writer.TryComplete(); try { c.Socket.Abort(); c.Socket.Dispose(); } catch { } } + _clients.Clear(); + } + try { _listener.Stop(); } catch { } + _listener.Close(); + _shutdown.Dispose(); + } + + // The whole viewer: one HTML file with inline CSS/JS, no external requests (CSP-safe, works offline). It lays + // the net out from the `topology` message, repaints on every `frame`, and — for people new to neural nets — + // explains each part on hover. The WebSocket auto-reconnects on drop so restarting a run re-attaches the viewer. + private const string Page = """ + + + + + +Network — live training + + + +
+

▚ live network

+ step — + loss — + eval — + ε — +
+ connecting… +
+
+ +
+
Hover any neuron, connection, or heatmap to learn what it means. Nodes are layers of neurons + (capped for display); edge brightness = weight magnitude. For dueling/value/policy nets the final small columns are + the network's output "heads".
+ +
+ + + + +"""; +}