diff --git a/.agents/rules/perf-work.md b/.agents/rules/perf-work.md new file mode 100644 index 00000000..24a0361e --- /dev/null +++ b/.agents/rules/perf-work.md @@ -0,0 +1,24 @@ +# Performance Work Rules (ENG2) + +Applies to any change under `**/rendering/**`, `**/graphics/**`, `**/benches/**`, `scripts/bench-gate.py`, or any issue labeled `area:performance`. + +## Definition of done (non-negotiable) + +A performance change is done only when BOTH hold: + +1. **A named scripted scene hits its numeric target** — S1–S5/H1/cull_scaling from `docs/src/runbook/perf-dod.md`, captured via `scripts/perf-capture` (3-run median on Apple M-series/Metal) or the CI `scripts/bench-gate.py` ratio-gate for CI-runnable benches. +2. **A phase counter attributes the cost.** The relevant GPU/CPU phase timing is non-zero and accounts for the frame. + +**A counter that reads zero means the instrumentation is broken, never that the work is free.** GPU cost surfaces at surface-acquire/present, not in a CPU command-record timer — measure it with a real GPU timestamp (QuerySet), do not infer it. + +## Rules + +- No optimization without a measurement first. Post the before-number, then the after-number, on the issue. +- Never silently cap coverage (top-N, sampling, windowing) to hit a number — `log`/document what was dropped. A scene-windowing workaround is a consumer band-aid, not an engine fix. +- The target is **O(visible), not O(total)**: per-frame CPU cost must not scale with total scene/entity count. If it does, that is the bug, regardless of FPS. +- Steady-state render frame (no scene mutation) must perform **0 heap allocations** in the render path — assert it with the alloc-budget harness (ENG2-P0-15), do not eyeball it. +- Aim for headroom, not the 60 FPS floor: the v2 targets are 120+ FPS for tens of thousands of visible entities. + +## Before closing + +Run the relevant bench-gate scene, attach the capture (scene, numbers, phase breakdown, delta vs baseline). See `docs/src/runbook/perf-dod.md` for scenes and targets and `[perf-work]`↔`[render-v2]`↔`[testing-v2]` for the companion rules. diff --git a/.agents/rules/render-v2.md b/.agents/rules/render-v2.md new file mode 100644 index 00000000..3b824ce2 --- /dev/null +++ b/.agents/rules/render-v2.md @@ -0,0 +1,25 @@ +# Render Core v2 Rules (ENG2 Phases 2 & 6) + +Applies to `**/graphics/**` and the renderer paths. These invariants supersede the performance/submission guidance in `graphics-patterns.md` (the backend-isolation and Renderer-trait rules there still hold). Full design: `docs/src/runbook/phases/phase-2.md` and RFC-0005. + +## Submission model + +- **O(visible), never O(total).** No per-frame iteration over the entire scene-object set. Cull against a spatial index (grid/BVH) *before* any transform/uniform/encode work. A culled object must pay zero per-frame cost. +- **Dense storage, not hash-map iteration.** Scene objects live in slot arrays with generational handles, iterated contiguously — not a per-frame walk of a `FxHashMap`. +- **Instancing by identity.** Draws are grouped by `(mesh id, material id)` and submitted instanced by default — including `CreatePlane`/`CreateCube`/`CreateSphere`/`CreateCylinder` primitives, not only `InstantiateModel`. A config flag that claims to enable instancing must actually drive the instanced path or not exist. + +## Uniforms & buffers + +- **Per-frame constants uploaded once** (camera, lights, fog) into one bind group. **Per-object data goes in dynamic-offset slots** — never copy a full multi-KB uniform block per draw, never issue one `write_buffer` per draw. Coalesce writes into one upload per frame. +- **Persistent, dirty-tracked instance/transform buffers.** Upload only changed ranges. Never rebuild the whole buffer, and never CPU-pre-transform vertices into a static VBO that must be re-baked on any change. +- Do not clone per-instance CPU vertex data for instances that share a mesh. + +## Data seam + +- Transforms are a **structure-of-arrays column shared between ECS and the renderer** (ENG2-P2-13). The renderer consumes dirty ranges from that column; it does not receive per-entity transforms across FFI one at a time. + +## Every render/UI/particle change + +- Add or update a **story** in the scene gallery (ENG2-P0-14) with a golden image and metric budget — see `[testing-v2]`. +- Meet the perf definition of done — see `[perf-work]` and `docs/src/runbook/perf-dod.md`. +- Raw GPU calls stay in `libs/graphics/backend/` (unchanged boundary). diff --git a/.agents/rules/testing-v2.md b/.agents/rules/testing-v2.md new file mode 100644 index 00000000..199d6726 --- /dev/null +++ b/.agents/rules/testing-v2.md @@ -0,0 +1,30 @@ +# Testing v2 Rules (ENG2) + +Applies to all ENG2 work. Full strategy: `docs/src/runbook/testing-strategy.md`. Extends `testing.md` (GL-context and unit conventions there still hold). + +## Spec-first (TDD) + +- Every behavioral issue gets a spec test at `goud_engine/tests/spec/eng2_p__.rs` encoding its Acceptance Criteria as executable assertions. The spec **is** the gate evidence. +- Write the spec RED before the implementation; make it GREEN; then REFACTOR. A reviewer confirms it was red-before-green. +- The issue's Verification section names the spec path. Do not close an issue whose spec test does not exist or does not assert its acceptance criteria. + +## Tiers — put each test at the right level + +- **Unit** — pure logic, no GPU. Design v2 render/ECS logic (culling, instancing-grouping, sort-key, propagation) to be unit-testable without a GPU context. +- **Isolation** — one subsystem behind its null/provider boundary (`NullBackend`, null net/audio, in-memory VFS). If the seam to mock a dependency is missing, add the null/mock provider. +- **Integration** — cross-layer in-process; assert exact draw/culled/visible counts (generalize `tests/renderer3d_frame_counts.rs`). +- **E2E** — SDK-driven headless engine (ENG2-P0-17); C#/TS/Python boot a real engine, run a scripted game, assert counts/world-hash/flat-RSS. + +## Story gallery + +- Every render/UI/particle/animation change adds or updates a **story** (ENG2-P0-14): scene + camera + knobs + expected invariants + golden image. CI renders it headless, diffs the golden, checks metric budgets, and publishes the HTML gallery. +- Changing a golden requires the documented golden-update approval — do not silently overwrite baselines. + +## Determinism, property, fuzz + +- Sim/scheduling changes run the determinism fixture (3 seeds × 10k ticks, hash-compare). Do not introduce nondeterminism (unordered iteration, wall-clock reads, FP drift). +- Component-store and command-decoder changes add/extend `proptest` + `cargo-fuzz` targets (ENG2-P0-18, P3-09). + +## Don't game the gate + +Never weaken an assertion, widen a tolerance, or skip a scene to make a gate pass. If a gate is wrong, fix the gate in the same PR with justification. diff --git a/.cursor/rules/perf-work.mdc b/.cursor/rules/perf-work.mdc new file mode 100644 index 00000000..b2687137 --- /dev/null +++ b/.cursor/rules/perf-work.mdc @@ -0,0 +1,31 @@ +--- +description: Performance Work Rules (ENG2) +globs: +alwaysApply: true +--- + + +# Performance Work Rules (ENG2) + +Applies to any change under `**/rendering/**`, `**/graphics/**`, `**/benches/**`, `scripts/bench-gate.py`, or any issue labeled `area:performance`. + +## Definition of done (non-negotiable) + +A performance change is done only when BOTH hold: + +1. **A named scripted scene hits its numeric target** — S1–S5/H1/cull_scaling from `docs/src/runbook/perf-dod.md`, captured via `scripts/perf-capture` (3-run median on Apple M-series/Metal) or the CI `scripts/bench-gate.py` ratio-gate for CI-runnable benches. +2. **A phase counter attributes the cost.** The relevant GPU/CPU phase timing is non-zero and accounts for the frame. + +**A counter that reads zero means the instrumentation is broken, never that the work is free.** GPU cost surfaces at surface-acquire/present, not in a CPU command-record timer — measure it with a real GPU timestamp (QuerySet), do not infer it. + +## Rules + +- No optimization without a measurement first. Post the before-number, then the after-number, on the issue. +- Never silently cap coverage (top-N, sampling, windowing) to hit a number — `log`/document what was dropped. A scene-windowing workaround is a consumer band-aid, not an engine fix. +- The target is **O(visible), not O(total)**: per-frame CPU cost must not scale with total scene/entity count. If it does, that is the bug, regardless of FPS. +- Steady-state render frame (no scene mutation) must perform **0 heap allocations** in the render path — assert it with the alloc-budget harness (ENG2-P0-15), do not eyeball it. +- Aim for headroom, not the 60 FPS floor: the v2 targets are 120+ FPS for tens of thousands of visible entities. + +## Before closing + +Run the relevant bench-gate scene, attach the capture (scene, numbers, phase breakdown, delta vs baseline). See `docs/src/runbook/perf-dod.md` for scenes and targets and `[perf-work]`↔`[render-v2]`↔`[testing-v2]` for the companion rules. diff --git a/.cursor/rules/render-v2.mdc b/.cursor/rules/render-v2.mdc new file mode 100644 index 00000000..9763f033 --- /dev/null +++ b/.cursor/rules/render-v2.mdc @@ -0,0 +1,32 @@ +--- +description: Render Core v2 Rules (ENG2 Phases 2 & 6) +globs: +alwaysApply: true +--- + + +# Render Core v2 Rules (ENG2 Phases 2 & 6) + +Applies to `**/graphics/**` and the renderer paths. These invariants supersede the performance/submission guidance in `graphics-patterns.md` (the backend-isolation and Renderer-trait rules there still hold). Full design: `docs/src/runbook/phases/phase-2.md` and RFC-0005. + +## Submission model + +- **O(visible), never O(total).** No per-frame iteration over the entire scene-object set. Cull against a spatial index (grid/BVH) *before* any transform/uniform/encode work. A culled object must pay zero per-frame cost. +- **Dense storage, not hash-map iteration.** Scene objects live in slot arrays with generational handles, iterated contiguously — not a per-frame walk of a `FxHashMap`. +- **Instancing by identity.** Draws are grouped by `(mesh id, material id)` and submitted instanced by default — including `CreatePlane`/`CreateCube`/`CreateSphere`/`CreateCylinder` primitives, not only `InstantiateModel`. A config flag that claims to enable instancing must actually drive the instanced path or not exist. + +## Uniforms & buffers + +- **Per-frame constants uploaded once** (camera, lights, fog) into one bind group. **Per-object data goes in dynamic-offset slots** — never copy a full multi-KB uniform block per draw, never issue one `write_buffer` per draw. Coalesce writes into one upload per frame. +- **Persistent, dirty-tracked instance/transform buffers.** Upload only changed ranges. Never rebuild the whole buffer, and never CPU-pre-transform vertices into a static VBO that must be re-baked on any change. +- Do not clone per-instance CPU vertex data for instances that share a mesh. + +## Data seam + +- Transforms are a **structure-of-arrays column shared between ECS and the renderer** (ENG2-P2-13). The renderer consumes dirty ranges from that column; it does not receive per-entity transforms across FFI one at a time. + +## Every render/UI/particle change + +- Add or update a **story** in the scene gallery (ENG2-P0-14) with a golden image and metric budget — see `[testing-v2]`. +- Meet the perf definition of done — see `[perf-work]` and `docs/src/runbook/perf-dod.md`. +- Raw GPU calls stay in `libs/graphics/backend/` (unchanged boundary). diff --git a/.cursor/rules/testing-v2.mdc b/.cursor/rules/testing-v2.mdc new file mode 100644 index 00000000..c2b9c1a9 --- /dev/null +++ b/.cursor/rules/testing-v2.mdc @@ -0,0 +1,37 @@ +--- +description: Testing v2 Rules (ENG2) +globs: +alwaysApply: true +--- + + +# Testing v2 Rules (ENG2) + +Applies to all ENG2 work. Full strategy: `docs/src/runbook/testing-strategy.md`. Extends `testing.md` (GL-context and unit conventions there still hold). + +## Spec-first (TDD) + +- Every behavioral issue gets a spec test at `goud_engine/tests/spec/eng2_p__.rs` encoding its Acceptance Criteria as executable assertions. The spec **is** the gate evidence. +- Write the spec RED before the implementation; make it GREEN; then REFACTOR. A reviewer confirms it was red-before-green. +- The issue's Verification section names the spec path. Do not close an issue whose spec test does not exist or does not assert its acceptance criteria. + +## Tiers — put each test at the right level + +- **Unit** — pure logic, no GPU. Design v2 render/ECS logic (culling, instancing-grouping, sort-key, propagation) to be unit-testable without a GPU context. +- **Isolation** — one subsystem behind its null/provider boundary (`NullBackend`, null net/audio, in-memory VFS). If the seam to mock a dependency is missing, add the null/mock provider. +- **Integration** — cross-layer in-process; assert exact draw/culled/visible counts (generalize `tests/renderer3d_frame_counts.rs`). +- **E2E** — SDK-driven headless engine (ENG2-P0-17); C#/TS/Python boot a real engine, run a scripted game, assert counts/world-hash/flat-RSS. + +## Story gallery + +- Every render/UI/particle/animation change adds or updates a **story** (ENG2-P0-14): scene + camera + knobs + expected invariants + golden image. CI renders it headless, diffs the golden, checks metric budgets, and publishes the HTML gallery. +- Changing a golden requires the documented golden-update approval — do not silently overwrite baselines. + +## Determinism, property, fuzz + +- Sim/scheduling changes run the determinism fixture (3 seeds × 10k ticks, hash-compare). Do not introduce nondeterminism (unordered iteration, wall-clock reads, FP drift). +- Component-store and command-decoder changes add/extend `proptest` + `cargo-fuzz` targets (ENG2-P0-18, P3-09). + +## Don't game the gate + +Never weaken an assertion, widen a tolerance, or skip a scene to make a gate pass. If a gate is wrong, fix the gate in the same PR with justification. diff --git a/AGENTS.md b/AGENTS.md index 6b596c6a..b95fafdd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -47,7 +47,7 @@ Source of truth: `.agents/agent-catalog.toml` and `.agents/role-specs/*.md`. ## Rules -Read only rules matching your change area: `.agents/rules/*.md`. +Read only rules matching your change area: `.agents/rules/*.md`. Active roadmap: v2 rebuild ("ENG2") — `docs/src/runbook/` + pinned master issue #810 (perf/render/test rules: `perf-work.md`, `render-v2.md`, `testing-v2.md`). ## Local AGENTS Files diff --git a/ALPHA_ROADMAP.md b/ALPHA_ROADMAP.md index bd7557fa..00d21422 100644 --- a/ALPHA_ROADMAP.md +++ b/ALPHA_ROADMAP.md @@ -1,12 +1,21 @@ -# GoudEngine Alpha Roadmap - -## Completion Status +# GoudEngine Alpha Roadmap (ARCHIVED) + +> **ARCHIVED — DO NOT USE FOR PLANNING.** This is the historical alpha (v1) roadmap. +> Its status claims are unreliable: the "Phase 0 (v2): Core Performance — COMPLETE" +> line below was false when written — #677, #678, and #679 were open priority:high +> perf bugs at the time, and the v2 perf gate (60 fps @ 10k entities) was never +> validated against a consumer-scale scene. +> +> **The current roadmap is the v2 Rebuild Runbook: `docs/src/runbook/phase-index.md` +> and the pinned master tracking issue (ENG2 #810).** Historical body preserved below. + +## Completion Status (v1 — historical) - [x] Phase 0 (v1): Foundation -- COMPLETE - [x] Phase 1 (v1): Core Stabilization -- COMPLETE - [x] Phase 2 (v1): Core Systems -- COMPLETE - [x] Phase 2.5 (v1): AI Debugger & Runtime Observability -- COMPLETE - [x] Phase 3 (v1): Rendering Pipeline + SDK Expansion -- COMPLETE -- [x] Phase 0 (v2): Core Performance (GoudEngine side) -- COMPLETE +- [ ] Phase 0 (v2): Core Performance -- SUPERSEDED by ENG2 (was falsely marked COMPLETE) ## Vision diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index a4be4ae2..43ce234c 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -26,6 +26,27 @@ - [Governance and Enforcement](development/governance.md) - [Performance Roadmap](development/performance-roadmap.md) +# v2 Rebuild Runbook (ENG2) + +- [Runbook Overview](runbook/index.md) +- [Phase Index](runbook/phase-index.md) +- [Perf Definition of Done](runbook/perf-dod.md) +- [Testing Strategy](runbook/testing-strategy.md) +- [Throne Sync](runbook/throne-sync.md) +- [Roadmap Issue Template](runbook/issue-template.md) +- [Phase Specs (full)](runbook/phase-specs.md) +- [Phase Specs — Perf/Test Expansion](runbook/phase-specs-expansion.md) +- [Phase 0: Instrumentation, Truth & Test Foundation](runbook/phases/phase-0.md) +- [Phase 1: SDK Single Source of Truth](runbook/phases/phase-1.md) +- [Phase 2: Render Core v2](runbook/phases/phase-2.md) +- [Phase 3: Data Core v2](runbook/phases/phase-3.md) +- [Phase 4: FFI v2 Convergence](runbook/phases/phase-4.md) +- [Phase 5: Parallelism & Determinism](runbook/phases/phase-5.md) +- [Phase 6: 2D Render v2](runbook/phases/phase-6.md) +- [Phase 7: Runtime Services](runbook/phases/phase-7.md) +- [Phase 8: Capability Gaps](runbook/phases/phase-8.md) +- [Phase 9: Authoring & Release](runbook/phases/phase-9.md) + # Architecture - [SDK-First Architecture](architecture/sdk-first.md) diff --git a/docs/src/runbook/index.md b/docs/src/runbook/index.md new file mode 100644 index 00000000..9f36b0d5 --- /dev/null +++ b/docs/src/runbook/index.md @@ -0,0 +1,50 @@ +# GoudEngine v2 Rebuild Runbook ("ENG2") + +This runbook is the authoritative plan for the GoudEngine v2 rebuild. It replaces the archived `ALPHA_ROADMAP.md`, whose "Phase 0 (v2): Core Performance — COMPLETE" claim was false when written (the perf gate was never validated against a consumer-scale scene, and #677/#678/#679 were open at the time). + +## Why this exists + +GoudEngine's flagship consumer, Throne (a C# colony-sim), ran at 1–17 FPS on moderate scenes and had to work around the engine (shadows disabled, an 80×80 terrain window, manual native-lib copying). A 12-subsystem, file:line-verified audit found two structural causes: + +1. **Render submission is O(total), per-object, un-instanced.** Every frame linearly scans the entire scene-object hash map (main + shadow pass); no spatial index; `CreatePlane`/`CreateCube` never instance; each draw re-uploads a full 4KB uniform block. +2. **The FFI data path takes 3 global mutexes per component op and the component store is implemented three times**; one heap malloc per component; a raw pointer escapes its mutex guard; transform propagation walks the full tree every frame; the engine is single-threaded by construction. + +The instrumentation that should have caught this was a facade (GPU timestamps disabled, so `shadow_pass` reported ~0 while frames stalled 300–600 ms; the perf-regression gate wired into zero CI jobs). + +## The goal + +Run **tens of thousands of visible entities at 120+ FPS** on Apple M-series and **100k+ simulation entities**, with per-frame cost **O(visible), not O(total)** — and become a general-purpose engine able to build any game. The forcing function is Throne, but the architecture targets far beyond it. + +## Binding decisions + +- **No legacy code kept** where we refactor. Breaking the API is allowed; Throne adopts afterward via filed `throne_ge` issues. +- **Full 10-SDK parity stays** — made cheap (one generated source of truth) and real (per-SDK CI gates), not pruned. +- **All platform targets stay** — made CI-real (Android/iOS gated, console experimental-gated), not pruned. Legacy OpenGL backend is removed per the no-legacy policy (ADR in Phase 0, executed in Phase 2). + +## Execution model + +- **Phases are sequential** (a phase's gate issue must close before the next starts) — except **Phase 1 ‖ Phase 2** run concurrently, and **Phases 5 ‖ 6 ‖ 7** run concurrently (their file sets are disjoint). +- **Batches within a phase are sequential; groups within a batch run in parallel** — one `/gh-issue --worktree` per group, separate agents/terminals. All groups in a batch merge before the next batch. +- **Every issue is self-contained** for `/gh-issue` execution (Architecture Context, Scope, Acceptance Criteria, Files Likely Touched, Agent Notes with file:line evidence, Verification). +- **Issues are disposable.** If an issue drifts from its phase doc, close it and regenerate from the runbook — never patch a drifted issue. +- **Every `breaking-change` issue names its Throne adoption follow-up** (in `throne_ge`, label `goudengine-adoption`) before it closes. + +## The perf definition of done + +See [perf-dod.md](perf-dod.md). A performance issue closes only when the named scripted scene (S1–S5) hits its numeric target **and** a phase counter attributes the cost. A counter that reads zero means broken instrumentation, never "free". + +## Where things live + +- **Phase index + gates:** [phase-index.md](phase-index.md) +- **Full phase specs:** [phases/phase-0.md](phases/phase-0.md) … `phase-9.md` +- **Perf scenes + targets:** [perf-dod.md](perf-dod.md) +- **Testing strategy (unit/isolation/integration/E2E/spec, story gallery, determinism/fuzz):** [testing-strategy.md](testing-strategy.md) +- **Throne adoption sync points:** [throne-sync.md](throne-sync.md) +- **Issue template:** [issue-template.md](issue-template.md) +- **Master tracking issue:** pinned in the GoudEngine repo (ENG2 master). + +## Concurrency map + +``` +P0 → (P1 ‖ P2) → P3 → P4 [SYNC A] → (P5 ‖ P6 ‖ P7) [SYNC B] → P8 [SYNC C] → P9 +``` diff --git a/docs/src/runbook/issue-template.md b/docs/src/runbook/issue-template.md new file mode 100644 index 00000000..5741394f --- /dev/null +++ b/docs/src/runbook/issue-template.md @@ -0,0 +1,52 @@ +## Parent +- **Program:** ENG2 — GoudEngine v2 Rebuild (see the pinned master tracking issue) +- **Phase / Milestone:** {phase name} (`{milestone}`) +- **Batch / Group:** {batch.group if known, else "see phase doc"} +- **Runbook spec:** `docs/src/runbook/phases/{phase-file}.md` (committed with the roadmap) + +## Summary +{1–3 sentences naming the exact artifact produced — concrete, not "improve X".} + +## Architecture Context +**Layer:** {Foundation | Libs | Services | Engine | FFI} (see `tools/lint_layers.rs` — downward-only deps). +**Modules/types touched:** +- `goud_engine/src/{path}` — {why} + +**Boundary constraints (only those that apply):** +- Raw GPU calls stay in `libs/graphics/backend/` — no `wgpu::` outside it. +- Generated `*.g.rs`/`*.g.cs`/`*.g.ts` are never hand-edited; change Rust + FFI + schema, rerun `./codegen.sh`. +- FFI exports: `#[no_mangle] extern "C"`, `#[repr(C)]`, null checks, `// SAFETY:` comments. + +**Pattern to follow:** {existing code to mirror, with path.} + +## Scope +- [ ] {deliverable 1} +- [ ] {deliverable 2} +- [ ] Tests: {spec test + unit/isolation as applicable} +- [ ] Docs/rules updates if this invalidates `.agents/rules/*` or `docs/src/*` + +## Acceptance Criteria +- [ ] {observable, testable} +- [ ] {PERF (if area:performance): numeric target on a named scene S1–S5 per `docs/src/runbook/perf-dod.md`, captured with non-zero phase-counter attribution} +- [ ] `cargo check && cargo fmt --all -- --check && cargo clippy -- -D warnings` clean; `cargo test` green; `./codegen.sh && git diff --exit-code` (drift gate) + +## Breaking Change & Throne Follow-up +{If breaking-change: name the changed/removed API, state NO compat shim is kept (user decision), and link the throne_ge adoption issue that must be filed before close. Else: "None — additive/internal."} + +## Blocked By +{list ENG2 IDs, e.g. "ENG2-P2-04, ENG2-P0-09" — or "None."} + +## Files Likely Touched +- **New/Modified/Generated:** {paths} + +## Agent Notes +{Non-obvious context with file:line evidence from the audit — prevents wrong assumptions. Include the specific evidence bullets from design-technical.md / design-expansion.md for this issue.} + +## Verification +```bash +cargo check && cargo fmt --all -- --check && cargo clippy -- -D warnings +cargo test +./codegen.sh && git diff --exit-code +# perf issues: python3 scripts/bench-gate.py (per perf-dod.md), scene capture attached +# spec test: goud_engine/tests/spec/{eng2_pN_nn_slug}.rs +``` diff --git a/docs/src/runbook/perf-dod.md b/docs/src/runbook/perf-dod.md new file mode 100644 index 00000000..8d159e38 --- /dev/null +++ b/docs/src/runbook/perf-dod.md @@ -0,0 +1,33 @@ +# Performance Definition of Done + +The failure mode this program exists to prevent: shipping a "perf" feature and closing its milestone without proving it against a consumer-scale scene — while the instrumentation that should have caught the regression reports zero. (That is exactly how sprite batching, materials/shadows, and metrics all shipped "done" while Throne still ran at 1–17 FPS.) + +## The rule + +A performance issue closes **only** when BOTH hold: + +1. **The named scripted scene hits its numeric target**, captured via `scripts/perf-capture` (ENG2-P0-07), 3-run median, on the Apple M-series dev machine (Metal) — or the CI ratio-gate (`scripts/bench-gate.py`) passes for CI-runnable benches. +2. **A phase counter attributes the cost.** The relevant `gpu_*`/CPU phase timing must be non-zero and account for the frame. **A counter that reads zero means broken instrumentation, never "free."** + +The closing PR posts the capture (scene, numbers, phase breakdown, bench-gate delta) on the issue. + +## Named scenes + +| Scene | Definition | Gate target (super-high-FPS) | +|---|---|---| +| **S1 throne-baseline** | ~6,400 terrain planes + ~1,431 model instances (~55k submeshes) + ~300 dynamic entities, shadows OFF (Throne field data) | BeginFrame+EndFrame ≤ **2 ms**; ≥ **120 FPS** (baseline today ≈ 42.5 ms / 15 FPS) | +| **S2 throne-target** | 200×200 map (40k tiles) as a tilemap primitive, 100k total scene objects, **30k visible**, 2k dynamic transforms/frame, shadows ON, 1 directional light | ≥ **120 FPS avg**; p99 ≤ **12 ms**; no frame > 20 ms | +| **S3 ffi-churn** | 100k entities × 4 components; per frame 50k reads, 10k writes, 1k spawn, 1k despawn via FFI | ≤ **2 FFI crossings**; ≤ **1 ms** engine-side; **0 steady-state heap allocs** | +| **S4 sprite-storm** | 2D: 100k sprites across 16 textures, **30k visible**, scrolling camera | ≥ **120 FPS**; ≤ **16 draws**; CPU submit ≤ **1 ms** | +| **S5 stress** | 500k objects / 50k visible | Informational nightly trend (self-hosted M-series runner, ENG2-P9-10); no crash, bounded RSS, frame documented — promoted to a hard gate in a later revision | +| **H1 headless-scale** | Headless: 100k and 1M entity tick, no GPU | 100k ≤ **3 ms**; 1M ≤ **16 ms** avg / **25 ms** max | +| **cull_scaling** | Fixed 5k visible, total 10k→100k, culling ON | frame-time growth ≤ **10%** | + +## Allocation & upload budgets (CI-gated via ENG2-P0-15) + +- Steady-state render frame (no scene mutation, D=0) performs **0 heap allocations** in the render path. +- Per-frame bytes-uploaded budget asserted per scene; the per-frame allocation inventory (material-sort clone, 4KB/draw uniform ring, per-instance vertex clones, UI string clones) is a tracked burndown checklist. + +## Why the numbers are credible + +Apple M-series sustains multi-GB/s buffer bandwidth and hundreds of thousands of instanced draws/frame under Metal when submission is GPU-friendly (persistent instance buffers, one draw per (mesh, material), per-frame constants uploaded once). 120 FPS is an 8.3 ms budget; the targets above leave real headroom rather than sitting on the 60 FPS floor. S5 is deliberately informational until the architecture proves out. diff --git a/docs/src/runbook/phase-index.md b/docs/src/runbook/phase-index.md new file mode 100644 index 00000000..866f1f25 --- /dev/null +++ b/docs/src/runbook/phase-index.md @@ -0,0 +1,42 @@ +# ENG2 Phase Index + +The GoudEngine v2 rebuild is 10 phases + a continuous lane. Full per-issue specs (with file:line evidence) are in [phase-specs.md](phase-specs.md); the perf-hardening + testing/infra delta is in [phase-specs-expansion.md](phase-specs-expansion.md). Live issue numbers are tracked in the pinned master issue (**ENG2 master #810**). + +| Phase | Milestone | Focus | Depends on | Runbook | +|---|---|---|---|---| +| 0 | `eng2-p0-truth` | Instrumentation, Truth & Test Foundation | — | [phase-0](phases/phase-0.md) | +| 1 | `eng2-p1-sdk-truth` | SDK Single Source of Truth | P0 gate | [phase-1](phases/phase-1.md) | +| 2 | `eng2-p2-render-core` | Render Core v2 (O(visible)) | P0 gate · ‖P1 | [phase-2](phases/phase-2.md) | +| 3 | `eng2-p3-data-core` | Data Core v2 (ECS + store) | P1 gate | [phase-3](phases/phase-3.md) | +| 4 | `eng2-p4-ffi-v2` | FFI v2 Convergence | P2 + P3 gates | [phase-4](phases/phase-4.md) | +| 5 | `eng2-p5-parallel` | Parallelism & Determinism | P4 gate | [phase-5](phases/phase-5.md) | +| 6 | `eng2-p6-render-2d` | 2D Render v2 | P4 gate · ‖P5,P7 | [phase-6](phases/phase-6.md) | +| 7 | `eng2-p7-services` | Runtime Services | P4 gate · ‖P5,P6 | [phase-7](phases/phase-7.md) | +| 8 | `eng2-p8-capabilities` | Capability Gaps | P5+P6+P7 gates | [phase-8](phases/phase-8.md) | +| 9 | `eng2-p9-authoring` | Authoring, Platforms, Examples | P8 gate | [phase-9](phases/phase-9.md) | +| — | `eng2-continuous` | Dependency health (no gate) | ongoing | — | + +## Concurrency + +``` +P0 → (P1 ‖ P2) → P3 → P4 [SYNC A] → (P5 ‖ P6 ‖ P7) [SYNC B] → P8 [SYNC C] → P9 +``` + +Within a phase: batches are sequential; groups within a batch run in parallel (`/gh-issue --worktree`, one per group). All groups in a batch merge before the next batch. + +## Gate summaries (super-high-FPS ambition — full checklists in each phase doc and [perf-dod.md](perf-dod.md)) + +- **P0** — GPU + CPU phase counters non-zero; `bench-gate.py` required in CI; scene harness S1–S5; story gallery; alloc-budget gate; E2E/spec/determinism/fuzz fixtures; Docker CI parity; RFC-0005 + OpenGL-removal ADR merged. +- **P1** — clean-room codegen green; compiled JNI symbols == manifest (167-method drift closed); per-SDK parity gate for all 10 SDKs; wasm coverage gated; C#/TS/Python integration tier in CI. +- **P2** — S1 BeginFrame+EndFrame ≤2 ms & ≥120 FPS; S2 (100k objects / 30k visible / shadows ON) ≥120 FPS avg, p99 ≤12 ms; 40k CreatePlane tiles ≤64 draws; cull_scaling ≤10%; #677/#678/#679 closed with captures. +- **P3** — zero global component statics; single lock per component op (bench-asserted); 100k-entity dirty propagation ≤0.5 ms; flat-RSS churn soak; pointer-escape fixed + fuzz green. +- **P4** — S3 per-frame op stream ≤2 FFI crossings, ≤1 ms engine-side, 0 steady-state allocs; fn-table entry point live; all 10 SDKs regenerate green. → **Sync A** (Throne). +- **P5** — H1: 100k tick ≤3 ms, 1M ≤16 ms avg/25 ms max; 3-seed determinism CI; parallel speedup ≥2.5×; frame limiter holds 60±1 FPS. → **Sync B**. +- **P6** — S4: 100k sprites / 30k visible ≥120 FPS, ≤16 draws, CPU submit ≤1 ms; exactly one sprite renderer remains. +- **P7** — async loads no frame >20 ms; mip chains + trilinear; one audio stack, one 2D physics; UI 1k-widget single-change relayout ≤0.3 ms; two windows. +- **P8** — nav 10k agents repath ≤5 ms; every capability ships FFI + 10-SDK parity + integration test + docs page; save/load round-trips to identical world hash; cgmath gone. → **Sync C**. +- **P9** — inspector edits + saves scene JSON; Android/iOS CI required; console typecheck lane; ≥2 example games build in CI; migration guide + agents/skills v2. + +## Named perf scenes + +Defined once (ENG2-P0-07), referenced by every perf gate — see [perf-dod.md](perf-dod.md): **S1** throne-baseline, **S2** throne-target (the prime-directive scene), **S3** ffi-churn, **S4** sprite-storm, **S5** stress (500k objects), **H1** headless-scale. diff --git a/docs/src/runbook/phase-specs-expansion.md b/docs/src/runbook/phase-specs-expansion.md new file mode 100644 index 00000000..dff5f3db --- /dev/null +++ b/docs/src/runbook/phase-specs-expansion.md @@ -0,0 +1,125 @@ +# ENG2 Roadmap Expansion — Perf Hardening + Testing/Infra (delta on design-technical.md) + +User directives driving this delta: +1. "tens of thousands of entities at SUPER HIGH fps" → raise ambition beyond 60fps parity to headroom (120+ FPS on Apple M-series for tens of thousands of visible entities; 100k+ sim). +2. Bake in TDD, spec-driven validation, unit/integration/E2E/isolation testing, a "Storybook for the engine", Docker/reproducible infra. + +This delta ADDS issues and RAISES gates. The base 91-issue structure in design-technical.md stands; below are the additions/changes. New engine total: **102 issues** (+11); Throne: **7** (unchanged). Grand total **109**. + +--- + +## A. Recalibrated performance gates (super-high-FPS ambition) + +Rationale: Apple M-series (M1/M2/M3) sustains multi-GB/s buffer bandwidth and hundreds of thousands of instanced draws/frame under Metal when submission is GPU-friendly. 60 FPS is a floor, not the goal. Targets stay physically credible (frame budget in ms) but demand real headroom. + +| Scene | OLD gate | NEW gate | +|---|---|---| +| **S1 throne-baseline** (55k objects / 10k visible, shadows off) | BeginFrame+EndFrame ≤ 4 ms; ≥ 60 FPS | **BeginFrame+EndFrame ≤ 2 ms; ≥ 120 FPS** | +| **S2 throne-target** (100k objects / **30k** visible, shadows ON, was 10k visible) | ≥ 60 FPS; p99 ≤ 33 ms | **≥ 120 FPS avg; p99 ≤ 12 ms; no frame > 20 ms** | +| **S3 ffi-churn** (100k entities; 50k reads/10k writes/1k spawn/1k despawn per frame) | ≤ 3 crossings; ≤ 2 ms engine-side | **≤ 2 crossings; ≤ 1 ms engine-side; 0 steady-state heap allocs** | +| **S4 sprite-storm** (100k sprites / **30k** visible) | ≥ 60 FPS; ≤ 20 draws | **≥ 120 FPS; ≤ 16 draws; CPU submit ≤ 1 ms** | +| **H1 headless-scale** (100k / 1M entity tick) | 100k ≤ 5 ms; 1M ≤ 33 ms | **100k ≤ 3 ms; 1M ≤ 16 ms avg / 25 ms max** | +| **S5 stress** (NEW: 500k objects / 50k visible) | — | **informational trend (nightly); no crash, no unbounded RSS, frame documented — becomes a hard gate in a later revision** | + +Add S5 to `perf-dod.md` scene table (defined in ENG2-P0-07). cull_scaling gate stays ≤ 10% frame growth for fixed-visible/growing-total. Every raised number updates the corresponding gate checklist in the phase doc; the acceptance criteria of ENG2-P2-12 (S1/S2 validation), ENG2-P4-08 (S3), ENG2-P5-06 (H1), ENG2-P6-08 (S4) change to the NEW numbers. + +--- + +## B. New performance issues + +### ENG2-P2-13 — SoA transform store shared between ECS and renderer (zero-copy extract) [L] · milestone p2 · area:rendering,area:ecs,area:performance,breaking-change +The ECS→renderer handoff is the expensive seam: today `SetModelPosition` pushes per-entity transforms across FFI, and `instantiate_model` clones vertices per instance (`core_model_instances.rs:39,77`), and the renderer holds transforms in `Object3D` inside a `FxHashMap`. Design a **structure-of-arrays transform column** (position/rotation/scale + world matrix) owned once, referenced by both ECS `GlobalTransform` reads and the renderer's per-instance buffer upload. Renderer consumes dirty ranges directly (no copy, no per-entity FFI). This is the backbone of the super-high-FPS target — it turns per-object work into per-dirty-range work. +Blocked by: ENG2-P2-01 (dense storage), ENG2-P0-09 (RFC-0005). Feeds ENG2-P2-05 (instance buffers), ENG2-P4-04 (bulk transform upload writes straight into this store). Throne follow-up: THR-A-02. +Agent notes: RFC-0005 (P0-09) must enumerate this handoff design; RFC-0006 (P3-01) must make the unified component store expose a columnar transform view compatible with this. Verify wgpu buffer-mapping strategy in `backend/wgpu_backend/init.rs`. + +### ENG2-P2-14 — GPU-driven culling / multi_draw_indirect path (Metal-gated stretch) [L] · milestone p2 · area:rendering,area:performance +Evaluate and, if Metal-viable under wgpu, implement indirect/compute-driven submission for the 30k+ visible target: a compute pass culls instances and writes a draw-indirect buffer, collapsing CPU submit cost toward O(1). Check `wgpu::Features::MULTI_DRAW_INDIRECT`/`INDIRECT_FIRST_INSTANCE` support on the Metal backend and what `init.rs` requests today. If Metal cannot support it acceptably, deliver the CPU spatial-index path (P2-02) as the shipped answer and document the indirect path as a future GPU-backend capability. RFC-0005 decision item. +Blocked by: ENG2-P2-02, ENG2-P2-04. Acceptance: S2 at 30k visible meets the 120-FPS gate with CPU submit ≤ 1 ms (whichever path achieves it). + +### ENG2-P5-07 — Per-frame frame-arena (bump) allocator + extract/prepare/submit split [L] · milestone p5 · area:memory,area:performance +Introduce a bump allocator reset each frame for all per-frame scratch (visible lists, draw-command buffers, extract staging) so steady-state frames do zero heap allocations. Adopt a bevy-style **extract → prepare → submit** split so sim state is snapshotted into render-owned buffers, enabling the parallel schedule (P5-02/P5-03) to overlap sim(N+1) with render(N). This is what lets the alloc-budget gate (P0-15) pass under load. +Blocked by: ENG2-P5-01 (Send), ENG2-P0-15 (alloc harness). Acceptance: S1 steady-state (D=0) render path = 0 heap allocations, asserted by the P0-15 counter; extract/submit overlap shows ≥ 1.5× throughput on the parallel bench. + +RFC scope edits (not new issues): +- **ENG2-P0-09 (RFC-0005)** decision list MUST add: CPU spatial index vs GPU compute culling; instanced draws vs multi_draw_indirect (Metal feature check); SoA transform handoff (P2-13); bind-group layout (per-frame/per-material/per-object dynamic offset); sort-key format. +- **ENG2-P3-01 (RFC-0006)** MUST require the store expose a columnar transform view usable by P2-13 with no copy. +- **ENG2-P4-01 (RFC-0007)** MUST specify bulk transform-upload ops writing directly into the P2-13 SoA store, and a 0-alloc decode path. + +--- + +## C. Testing & validation workstream (new Phase 0 issues — regression net BEFORE the render rewrite) + +These land in Phase 0 deliberately: you cannot safely rewrite the render core (P2) or unify the store (P3) without a spec/story/determinism net that catches regressions. All are `area:testing`/`area:infra`. + +### ENG2-P0-13 — Docker CI image + `docker compose verify` local/CI parity [M] · milestone p0 · area:infra +Audit the existing root `Dockerfile` + `docker-compose.yml` (verify what they do today). Build a versioned `Dockerfile.ci`: pinned `rust-toolchain.toml` toolchain + Mesa lavapipe/llvmpipe (for headless GPU + story rendering) + SDK toolchains (dotnet/node/python). Publish to GHCR; cache via sccache or cargo-chef layers. A `docker compose run verify` entrypoint runs the SAME `scripts/verify.sh` gate as CI, so local == CI. Document the Metal-only exception (real-GPU perf capture stays on the M-series host per perf-dod). +Acceptance: `docker compose run verify` reproduces the CI core gate result; image pinned + published; `scripts/check-gate-parity.py` extended to include the container entrypoint. + +### ENG2-P0-14 — Scene story gallery MVP ("Storybook for the engine") [L] · milestone p0 · area:testing,area:rendering +Build `goud_gallery`: a declarative **story** = scene setup + camera + knobs (args) + expected invariants (draw calls, instance counts, phase-timing budgets) + golden-image tag. A headless runner renders each story to a texture on the wgpu backend (no window), captures framebuffer + metrics via the existing MCP `capture_frame`/`get_metrics_trace`, perceptually diffs against a committed golden (evaluate `dssim`/`image-compare`), and checks metric budgets. CI emits a static **HTML gallery** artifact per PR (image, diff-vs-golden, metrics table, pass/fail) — the Storybook browsing experience. Reuse `benches/helpers/scene3d.rs` scene builders and the `tests/renderer3d_frame_counts.rs` count-pinning pattern. MVP catalog: a handful of core stories (sprite batch, one instancing path, one shadow config); every later render/UI/particle/animation issue ADDS stories as an acceptance item. +Acceptance: gallery runs on lavapipe in CI + real Metal locally with per-backend tolerance; golden-update protocol documented (who approves a changed golden; baselines versioned in-repo, small PNGs, or LFS if size demands). Interaction stories (scripted input via MCP `inject_input`/`replay`, assert end-state) supported as the "play function" analog. + +### ENG2-P0-15 — Allocation-budget harness + steady-state zero-alloc gate [M] · milestone p0 · area:testing,area:performance +Add a global-allocator counter (custom `GlobalAlloc` wrapper or `dhat` feature) usable in benches/tests to assert allocations-per-frame and bytes-uploaded-per-frame budgets. Wire into `bench-gate.py` so budgets are CI-gated. Establish the contract: **steady-state render frame (no scene mutation, D=0) performs 0 heap allocations**; document current baseline (expected-fail today, becomes gated as P2/P5 land). Inventory the per-frame allocation sites (audit found: material sort clone `render/mod.rs:264-282`, uniform ring 4KB/draw, per-instance vertex clones, UI string clones per frame, `finalize_batch` to_vec) as the burndown checklist. +Acceptance: alloc counter available to benches; budget assertions in CI; the per-frame-allocation inventory committed as a tracked checklist. + +### ENG2-P0-16 — Spec-test convention + TDD gate [M] · milestone p0 · area:testing +Define the issue→spec mapping: each ENG2 issue with observable behavior gets a spec test at `goud_engine/tests/spec/eng2_p__.rs`, tagged, that encodes its Acceptance Criteria as executable assertions and is the gate evidence. Extend the `tdd-workflow` skill for RED-GREEN-REFACTOR against spec tests; add to the issue template's Verification section a required "spec test path" line; add a reviewer check (spec-reviewer) that the spec exists and was red-before-green. A CI job asserts every closed `roadmap:eng2` issue with `area:performance`/behavioral scope has a corresponding spec file. +Acceptance: convention documented in the runbook; template updated; `tdd-workflow` skill extended; example spec test committed for one Phase 0 issue. + +### ENG2-P0-17 — E2E SDK-driven headless harness (C#/TS/Python) [L] · milestone p0 · area:testing,area:sdk-csharp +A harness where each first-class SDK boots a REAL headless engine, runs a scripted mini-game N frames, and asserts draw counts / world-state hash / no-crash / flat RSS. Lives under `sdks/*/e2e/` + a shared scene script format. Runs in the Docker CI image (P0-13) on lavapipe. This is the missing tier: today secondary SDK tests never touch a running engine (Swift EnumTests 36 LOC), and `validate_coverage.py` checks only manifest parity. Minimum: C#, TypeScript, Python; other SDKs keep scaffolds + the P1-04 parity gate. +Acceptance: three SDKs run a scripted headless game in required CI; assertions include a cross-SDK identical world-state hash (ties to determinism P5-04) once available. + +### ENG2-P0-18 — Determinism, property & fuzz fixtures [M] · milestone p0 · area:testing +Reusable fixtures: (a) determinism harness — run the headless sim 3 seeds × 10k ticks, compare world hashes (fixture consumed by P5-04 and Throne's determinism gate); (b) `proptest` generators for culling (visible-set correctness vs brute force), the command-buffer decoder (P4-02), and instancing grouping; (c) `cargo-fuzz` targets for the component store add/get/remove/batch and the command-buffer decoder (P4). Nightly CI lane runs fuzz with a time budget. +Acceptance: determinism fixture + at least one proptest + one fuzz target running in nightly CI, zero findings; documented so P3/P4/P5 issues plug into them. + +### ENG2-P0-19 — CI restructure: tiered gate + path filters + merge queue [M] · milestone p0 · area:infra +Today ~20 jobs (incl. 10-SDK fanout) gate every merge; shipping one fix is slow. Restructure into: **fast core gate** (fmt, clippy -D warnings, unit + isolation tests, lint-layers) always-on, < 10 min; **staged tiers** (integration, E2E P0-17, story gallery P0-14, blocking GPU lane P0-06, SDK matrix) triggered by path filters + a merge queue. New Phase 0 required gates (bench-gate P0-05, GPU lane P0-06, parity P1-04) slot into the right tier without doubling wall-clock. Coordinates with release.yml decoupling (P1-07). +Acceptance: core gate < 10 min p50; heavy tiers path-filtered; merge queue enabled; documented job graph. + +### Phase 0 gate additions (append to the P0 gate checklist) +- [ ] `docker compose run verify` == CI core gate result. +- [ ] Story gallery runs in CI with ≥ 5 committed golden stories; HTML gallery artifact published per PR. +- [ ] Alloc-budget counter available and gated; per-frame allocation inventory committed. +- [ ] Spec-test convention live; issue template + tdd-workflow updated; ≥ 1 example spec committed. +- [ ] E2E harness boots C#/TS/Python against a headless engine in required CI. +- [ ] Determinism + proptest + fuzz fixtures running (nightly for fuzz), zero findings. +- [ ] CI core gate < 10 min; tiered graph + merge queue live. + +--- + +## D. New infra issue in Phase 9 + +### ENG2-P9-10 — Self-hosted M-series nightly perf runner + trend dashboard [M] · milestone p9 · area:infra,area:performance +GitHub-hosted runners are too noisy for absolute perf numbers and can't run Metal. Register the M-series dev machine as a labeled self-hosted runner (security-scoped: nightly-only, no fork PRs). Nightly pipeline runs scenes S1–S5 + benches on real Metal, captures via perf-capture (P0-07), publishes a trend dashboard (simple JSON → Pages chart) and alerts on regression beyond the bench-gate ratio threshold. This is where the "super high FPS" numbers are actually measured and defended over time. +Acceptance: nightly job runs on the self-hosted M-series runner; S1–S5 trend chart published; regression alert fires on a seeded regression test. + +Phase 9 gate addition: +- [ ] Nightly M-series perf runner live; S1–S5 trend dashboard published; regression alerting verified. + +--- + +## E. Revised issue-count table + +| Phase | Milestone | Engine issues | Δ | Throne | +|---|---|---|---|---| +| P0 Instrumentation, Truth & Test Foundation | eng2-p0-truth (#10) | 19 | +7 | 1 (Sync 0) | +| P1 SDK Source of Truth | eng2-p1-sdk-truth (#11) | 10 | — | — | +| P2 Render Core v2 | eng2-p2-render-core (#12) | 14 | +2 | — | +| P3 Data Core v2 | eng2-p3-data-core (#13) | 9 | — | — | +| P4 FFI v2 Convergence | eng2-p4-ffi-v2 (#14) | 8 | — | 4 (Sync A) | +| P5 Parallelism & Determinism | eng2-p5-parallel (#15) | 7 | +1 | 1 (Sync B) | +| P6 2D Render v2 | eng2-p6-render-2d (#16) | 8 | — | — | +| P7 Runtime Services | eng2-p7-services (#17) | 6 | — | — | +| P8 Capability Gaps | eng2-p8-capabilities (#18) | 11 | — | 1 (Sync C) | +| P9 Authoring/Platforms/Polish | eng2-p9-authoring (#19) | 10 | +1 | — | +| Continuous | eng2-continuous (#20) | (carried #704/#657 + dep-dedup lives in P1-10) | — | — | +| **Total** | | **102** | **+11** | **7** → **109** | + +## F. Runbook doc deltas +- `perf-dod.md`: add S5 scene; raise all gate numbers per section A; add the alloc/frame + bytes-uploaded/frame budget definitions. +- New runbook page `testing-strategy.md`: the five tiers (unit/isolation/integration/E2E/spec) + story gallery + determinism/property/fuzz, with the issue→spec mapping rule. +- Issue template Verification section: add required "Spec test:" line and "Stories added/updated:" line for render/UI issues. +- New `.agents/rules/testing-v2.md`: the spec-test convention + story-gallery authoring as agent law; extend the `tdd-workflow` and add a `story-authoring` skill. diff --git a/docs/src/runbook/phase-specs.md b/docs/src/runbook/phase-specs.md new file mode 100644 index 00000000..77e1f92a --- /dev/null +++ b/docs/src/runbook/phase-specs.md @@ -0,0 +1,617 @@ + +# GoudEngine v2 Technical Roadmap — Phases, Batches, Groups, Gates, and Issue Decomposition + +**Status:** Design for approval. On approval this becomes `docs/runbook/v2/phase-index.md` + `docs/runbook/v2/phases/phase-N.md` in aram-devdocs/GoudEngine, one GitHub milestone per phase, one issue per ENG2 ID below, a new master tracking issue (superseding #475/#114), and 7 issues in aram-devdocs/throne_ge at the marked sync points. + +**Prime directive (from binding user decisions):** Throne (colony-sim, C#) at 60 fps with 10k+ visible entities and 100k+ sim entities; engine per-frame cost O(visible), not O(total). Full 10-SDK parity kept but made cheap and real. No legacy code retained where we refactor; breaking changes allowed, Throne adopted afterward via filed throne_ge issues. All platform targets kept and made CI-real. + +**Total issue count: 98** — 91 GoudEngine issues across 10 phases + 7 throne_ge adoption issues at 4 sync points. (Within the 60–100 target.) + +--- + +## 1. Structure and Concurrency Model + +The runbook mirrors the throne_ge convention (`docs/runbook/phase-index.md`): **phases → sequential batches → parallel groups**, all-merge-before-next-batch, per-issue self-contained specs (Architecture Context / Scope checklist / Acceptance Criteria / Files Likely Touched / Agent Notes with file:line evidence), and explicit `/gh-issue --worktree` run commands per batch. + +One deliberate extension over the throne_ge convention: **two concurrency-eligible pairs of phases**, because genuine parallelism exists between file-disjoint workstreams and forcing a linear chain would idle agent capacity: + +``` +Phase 0 Instrumentation & Truth (W0 + W8-docs + RFCs) ── everything blocks on this + │ + ├── Phase 1 SDK Single Source of Truth (W6-core) ─┐ CONCURRENT PAIR A + └── Phase 2 Render Core v2 (W1) ─┘ (disjoint: codegen/ sdks/ src/jni src/wasm vs libs/graphics/ ffi/renderer3d/) + │ + Phase 3 Data Core v2 (ECS unification) (W2-store) — requires Phase 1 gate; MAY overlap Phase 2 Batches 2.2+ + │ (disjoint: ecs/ ffi/component* context_registry vs render files) + Phase 4 FFI v2 Convergence (W2-ffi) — requires Phase 2 AND Phase 3 gates + │ ══ SYNC A: Throne adoption (4 throne_ge issues) — the prime-directive validation ══ + │ + ├── Phase 5 Parallelism & Determinism (W3) ─┐ + ├── Phase 6 2D Render v2 (W4) ─┼─ CONCURRENT TRIPLE B (pairwise-disjoint: + └── Phase 7 Runtime Services (W5) ─┘ ecs/schedule vs 2D render stack vs assets/ui/audio/platform) + │ ══ SYNC B: Throne job-system/determinism adoption (1 throne_ge issue) ══ + │ + Phase 8 Capability Gaps (W7) — requires 5+6+7 (nav needs jobs+spatial; 2D particles need sprite core; streaming needs async pipeline) + │ ══ SYNC C: Throne capability evaluation (1 throne_ge issue) ══ + │ + Phase 9 Authoring, Platforms, Examples (W8 + W6-platform) +``` + +Sequencing decision for the W1/W2 overlap (required by the task): **render core (Phase 2) starts first and owns the critical path**; data core (Phase 3) starts as soon as the SDK-truth gate (Phase 1) passes — codegen must be healthy *before* any FFI-surface rewrites, or every W2 deliverable widens the 167-method JNI drift and the 62/682 wasm gap. The two touchpoint items that need both (bulk transform upload, command-buffer draining renderer ops) are deliberately pulled out of both phases into **Phase 4: Convergence**. + +**Within a concurrency pair, each phase still runs its own batches sequentially.** No batch in any phase starts until Phase 0's gate passes. + +### Issue conventions +- **ID scheme:** `ENG2-P-` (GoudEngine), `THR--` (throne_ge). GitHub titles: `ENG2-P2-08: Auto-instance primitives by (mesh,material) identity`. +- **Effort:** S ≈ small focused PR, M ≈ one full `/gh-issue` agent run, L ≈ one large run or two chained PRs. No XL issues remain (all split). +- **Milestones:** `v2-phase-0` … `v2-phase-9` in GoudEngine. Labels: existing `type:*`/`area:*`/`priority:*` plus new `phase:v2-p{0..9}`; legacy label dedup handled in P0-12. +- **Every breaking-change issue names its Throne adoption follow-up** in its body (the THR issue ID from the sync tables below). +- **Issue body template (all issues):** Architecture Context → Scope (checklist) → Acceptance Criteria (numeric where perf) → Files Likely Touched → Agent Notes (the file:line evidence bullets below go here verbatim). + +### Perf gate measurement procedure +CI runners cannot measure Metal. Two enforcement layers: +1. **CI (blocking):** `scripts/bench-gate.py` ratio-normalized regression gate on the tracked bench set (P0-05); blocking lavapipe GPU-test lane (P0-06); count-pinning tests (draw calls, culled counts). +2. **Phase gate sign-off (manual, scripted):** `scripts/perf-capture` harness (P0-07) runs the named scene on the Apple M-series dev machine, 3-run median, and the capture report is posted on the phase tracking issue before the gate box is checked. + +### Named scripted scenes (defined once, in P0-07; referenced by every perf gate) + +| Scene | Definition | Purpose | +|---|---|---| +| **S1 throne-baseline** | Replica of Throne field data: 6,400 terrain planes + 1,431 model instances (~55k submeshes) + 300 dynamic entities, shadows OFF (per `profiling/profile-GameWorld-20260420-224800.md` + throne_ge `Game3DSceneRenderer.Terrain.cs:8-16`) | Regression anchor; today: BeginFrame 21.7 ms + EndFrame 20.8 ms + 41 ms overhead ≈ 15 FPS | +| **S2 throne-target** | 200×200 map (40k tiles) as tilemap primitive, 100k total scene objects, 10k visible, 2k dynamic transform updates/frame, shadows ON, 1 directional light | The prime-directive scene | +| **S3 ffi-churn** | 100k entities × 4 components; per frame: 50k component reads, 10k writes, 1k spawns, 1k despawns via FFI | Data/FFI core gate | +| **S4 sprite-storm** | 2D: 100k sprites across 16 textures, 10k visible, scrolling camera | 2D core gate | +| **H1 headless-scale** | Headless: 100k and 1M entity tick, no GPU | Parallelism/determinism gate; feeds Throne Phase 1 | + +--- + +## 2. Phase 0 — Instrumentation & Truth (W0) + +**Goal:** Measure before optimizing. Every later perf claim must be attributable to a non-zero phase counter and gated by a bench. Decision records for the two big redesigns are written here with fresh data. 12 issues. + +### Batch 0.1 — Instrumentation core (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P0-01 | Add GPU timestamp queries (wgpu QuerySet) for shadow/render/submit phases | A | M | — | +| ENG2-P0-02 | Fix always-zero CPU phase counters and surface both timing families via FFI + MCP | A | M | P0-01 | +| ENG2-P0-03 | Bench suite expansion: cull_scaling, primitive draw-call, real-GPU shadow benches | B | L | — | +| ENG2-P0-04 | Feature-gated tracy/puffin profiler integration | C | M | — | + +- **P0-01:** `timestamp_writes: None` at `backend/wgpu_backend/frame.rs:148` and `shadow_pass.rs:76,382` — no GPU timing exists; all phases are `Instant::now()` CPU record-time (`frame.rs:18-127`). Add QuerySet begin/end around shadow, main pass, submit; new `gpu_shadow/gpu_render/gpu_submit` phases distinct from CPU-record phases. This is why shadow_pass reports 0 µs while Metal stalls 300–600 ms (#677 evidence). +- **P0-02:** Consumer profile shows `uniform_upload, render_pass, gpu_submit, surface_present, shadow_build, bone_*` all 0 µs (throne profile :44-52). Wire real values through `frame_timing.rs` → `GetFramePhaseTimings` FFI → MCP `get_metrics_trace` (tools/goudengine-mcp inherits the mis-attribution today). +- **P0-03:** `benches/helpers/scene3d.rs:81` force-disables frustum culling in every bench scene; `renderer3d_frame_benchmarks.rs:29,35` scale only total count — bug #678 is structurally unmeasurable. Add: (a) `cull_scaling` group — fixed 5k visible, total 10k→100k, culling ON; (b) draw-call bench for `CreatePlane`/`CreateCube` (count-pinning test `tests/renderer3d_frame_counts.rs` already documents draw_calls == n); (c) opt-in real-wgpu shadow bench (NullBackend at `helpers/null_backend.rs:46-53` no-ops everything, so `shadow_record/casters_*` measure the wrong thing). +- **P0-04:** No tracy/puffin in any Cargo.toml. Feature-gate `profiling` crate spans in the frame path + ECS schedule so devs get flamegraphs beyond the 14 fixed counters. + +### Batch 0.2 — CI wiring & scene harness (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P0-05 | Wire bench-gate.py into required CI; fix benchmarks.yml bench set; retire bench-gate.sh (absorbs #256) | A | M | P0-03 | +| ENG2-P0-06 | Promote curated GPU tests to a blocking lavapipe lane | B | M | — | +| ENG2-P0-07 | Scripted consumer-scale scene harness: S1/S2/S3/S4/H1 + `scripts/perf-capture` + committed S1 baseline report | C | L | P0-01, P0-02 | +| ENG2-P0-08 | Move community-stats bot off protected main; revoke admin push token | D | S | — | + +- **P0-05:** `scripts/bench-gate.py` (ratio-normalized, baselines at `benches/baselines/criterion_baseline.json`) has ZERO references in `.github/`/`.husky/`/`verify.sh`; `benchmarks.yml:35` runs the wrong bench set, weekly-cron only, comparing nothing. Make it a required PR check over engine_tick + renderer3d_frame + the new P0-03 benches; delete redundant `bench-gate.sh`. +- **P0-06:** 96 `#[ignore]` GPU tests run only in `headless-gpu.yml` with `continue-on-error: true` (line 25) — renderer is untested in gating CI (`ci.yml:174-178` runs only `null`-named tests). Promote a curated subset to required. +- **P0-07:** Builds the five named scenes (table above) as reproducible harness scenarios (headless + capture via MCP `capture_frame`/`get_metrics_trace`), documents the 3-run-median M-series procedure, and commits the S1 baseline numbers as the regression anchor for the Phase 2 gate. +- **P0-08:** `community-stats.yml:5-6,20-25,40-41` — daily cron force-pushes `[skip ci]` commits to protected main with an admin token (122 commits pollute blame/bisect for perf work). Move to a data branch/Pages; drop the token. + +### Batch 0.3 — Decision records & repo truth (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P0-09 | RFC-0005: Render submission redesign (spatial index, dense storage, instancing, uniform strategy) | A | M | P0-07 | +| ENG2-P0-10 | ADR: Remove legacy glfw/OpenGL backend; verify live backend; reconcile doc contradiction | A | S | — | +| ENG2-P0-11 | Docs truth pass: archive ALPHA_ROADMAP.md, promote performance-roadmap.md, fix ARCHITECTURE.md, adopt perf definition-of-done | B | M | — | +| ENG2-P0-12 | Issue/label/milestone hygiene + verify-close pass (#423, #670, #136, #114, #475; migrate #590–593) | C | M | — | + +- **P0-09:** No RFC covers the redesign #677/#678/#679 require (docs/rfcs has only RFC-0001..0004). Must decide: grid vs BVH; dense-slot storage replacing `FxHashMap` (`renderer3d/core/mod.rs:54-93`); dynamic-offset uniform slots; instancing-by-identity keying. Written with fresh P0-07 baseline numbers. Blocks Phase 2 Batch 2.1. +- **P0-10:** ARCHITECTURE.md says wgpu default; ALPHA_ROADMAP.md:1246-1248 says wgpu incomplete/GL bypasses abstraction. User decision: "no legacy code kept" — record removal of the legacy-glfw-opengl path as an ADR (execution is ENG2-P2-03). Confirm the active Metal path in code first. +- **P0-11:** ALPHA_ROADMAP.md:9 falsely claims "Phase 0 (v2) Core Performance — COMPLETE" while #677/#678/#679 are open. Archive it; promote `docs/src/development/performance-roadmap.md` (current, code-verified); write the perf definition-of-done ("closed only when the named scene hits the number AND the phase counter attributes the cost") into CLAUDE.md/agents + PR template. +- **P0-12:** Dedupe legacy labels (bug/enhancement/…) into `type:*`; add `phase:v2-p*`; close milestones 1–9 as superseded; retag #677-679 `area:performance`; verify-close: #423 (drawText EXISTS in wasm — `wasm/rendering.rs:239`, close invalid), #670 (`sdks/csharp/build/GoudEngine.targets` exists and packs — fix its repo-root path-preference fallback, then close), #136 (debugger substantially shipped — close, successor is P9-01), #114/#475 (superseded by new master issue); migrate #590–593 bodies to throne_ge and close GE-side. + +### Phase 0 Gate +- [ ] S1 capture: `gpu_shadow`, `gpu_render`, `gpu_submit` report non-zero; CPU counters `uniform_upload/render_pass/surface_present` non-zero; MCP `get_metrics_trace` shows both families. +- [ ] bench-gate.py is a required PR check over the tracked set (engine_tick, renderer3d_frame, cull_scaling, draw-call, ffi_component); baselines committed. +- [ ] cull_scaling baseline recorded and documents the current O(total) defect (expected-fail documented, becomes gated in Phase 2). +- [ ] Blocking lavapipe lane required on PRs; curated GPU tests pass. +- [ ] S1 baseline report committed (re-measured post-#670-fix numbers). +- [ ] RFC-0005 and the OpenGL-removal ADR merged; ALPHA_ROADMAP.md archived; zombie milestones closed. + +**Sync 0 (throne_ge, after gate):** `THR-S0-01` [S] — bump GoudEngine, delete the `CopyGoudEngineNativeLib` workaround (`Throne.App.csproj:33-49`), confirm the ~60x load-path overhead is gone, re-profile GameWorld with the fixed phase counters and post the report (this becomes Phase 2's consumer-side "before" evidence). Maps throne_ge #1040 partially. + +--- + +## 3. Phase 1 — SDK Single Source of Truth (W6-core) — *concurrent with Phase 2* + +**Goal:** Parity across 10 SDKs becomes cheap (one generated source of truth) and real (CI-gated per SDK, integration-tested). Must gate before Phase 3 so FFI rewrites propagate cleanly. 10 issues. + +### Batch 1.1 — Fix the broken generators (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P1-01 | Fix JNI codegen dual-file bug; add CI assert compiled-JNI-symbols == manifest | A | L | — | +| ENG2-P1-02 | Upgrade jni crate 0.21 → 0.22 (#657) | B | S | — | + +- **P1-01:** `jni/mod.rs:10-11` compiles ONLY `generated.g.rs` (421 symbols, frozen 2026-04-02) while `codegen/gen_jni.py:21,2275` writes orphaned `generated.rs` (588 symbols) — Kotlin/Android ships missing 167 methods incl. all frame-timing FFI. Point generator at the compiled file, delete the orphan, fix `scripts/check-rs-line-limit.sh:25` + `tools/lint_layers.rs:286` which reference the dead file, and add the CI symbol-parity assert. +- **P1-02:** Straight dependency upgrade; do after P1-01 so regen is trustworthy. + +### Batch 1.2 — Single source of truth (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P1-03 | Generate ffi_mapping.json from the Rust manifest; delete the 30-entry `_TYPE_ALIASES` drift table | A | M | — | +| ENG2-P1-04 | Per-SDK export-parity CI gate: diff each generated SDK against the manifest | A | M | P1-03 | +| ENG2-P1-05 | Bring wasm/web bridge under manifest codegen + coverage gate; wasm-pack modernization (#425) | B | L | P1-03 | + +- **P1-03:** Today three sources: auto-extracted `ffi_manifest.json` (669 fns) vs hand-maintained `ffi_mapping.json` + 430KB `goud_sdk.schema.json`; `validate_coverage.py` papers over drift with a hardcoded 30-entry alias map. Normalize types at extraction, make the manifest the sole root. +- **P1-04:** `validate_coverage.py` never inspects generated SDK outputs — 671 C# DllImports vs 669 manifest fns is ungated. Promote the `sdk-parity-check` skill concept into `codegen.sh` + required CI for all 10 SDKs. +- **P1-05:** `src/wasm/*.rs` (3.9k LOC) is hand-written, 62/682 exports covered; `gen_ts_web.py` already exists. Generate the wasm bridge from the manifest, enforce coverage (explicit exclusion list for GPU-inapplicable exports), and replace deprecated wasm-pack tooling (#425). + +### Batch 1.3 — SDK reality & release cost (5 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P1-06 | Engine-integration test tier: C#/TS/Python E2E against headless engine (absorbs #137) | A | L | — | +| ENG2-P1-07 | Decouple 9-registry publishing from patch bumps; per-ecosystem dispatch | B | M | — | +| ENG2-P1-08 | Make the Rust SDK real (replace 10-line published stub) | C | M | — | +| ENG2-P1-09 | Decision + execute: #[goud_api] macros as generator of no_mangle surface, or archive | D | M | P1-03 | +| ENG2-P1-10 | Dependency health: #704 security deps + RUSTSEC ignored-advisory burndown + dep dedup | E | M | — | + +- **P1-06:** Secondary SDK tests are scaffold-only (Swift EnumTests 36 LOC; Kotlin type tests) — never touch a running engine. Add an integration tier (headless engine, spawn/query/draw-count assertions) for C#/TS/Python minimum; other SDKs keep scaffolds + parity gate. +- **P1-07:** `release.yml` is 801 lines publishing to 9 registries on every patch (version 0.0.841); `release-please-config.json` pins 20 files. Publish only on explicit tag/dispatch per ecosystem. +- **P1-08:** `sdks/rust` is a 10-line lib.rs published to crates.io. Wire it to the real native `sdk/` module surface with tests, or re-export the engine crate SDK — no stub publishing. +- **P1-09:** `goud_engine_macros` `#[goud_api]` is a third codegen mechanism across 21 files. Decision record + execution: either it *generates* the `#[no_mangle]` surface feeding the manifest (reducing hand-written FFI), or it's archived. Cannot stay ambiguous into Phase 4's fn-table work. +- **P1-10:** 8 ignored RUSTSEC advisories in deny.toml incl. anyhow UB; #704's dependency list; duplicate crate versions. + +### Phase 1 Gate +- [ ] Clean-room `./codegen.sh` green; compiled JNI symbol set == manifest (drift = 0, was 167). +- [ ] Per-SDK parity gate required in CI for all 10 SDKs; C# DllImport count == manifest count. +- [ ] wasm/web coverage == manifest minus documented exclusions, CI-gated (was 62/682). +- [ ] C#/TS/Python integration tests run a headless engine in required CI. +- [ ] A patch version bump publishes zero registries without explicit dispatch. +- [ ] RUSTSEC ignore list ≤ 2 entries, each with a written justification. + +--- + +## 4. Phase 2 — Render Core v2 (W1) — *concurrent with Phase 1* + +**Goal:** Per-frame cost O(visible); instancing by default; shadows usable. Closes #677/#678/#679. This is the highest-impact phase in the roadmap. 12 issues. + +### Batch 2.1 — Foundations from RFC-0005 (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-01 | Dense scene storage: slot arrays + generational handles replacing FxHashMap iteration | A | L | P0-09 | +| ENG2-P2-02 | Scene spatial index; move culling upstream of encode so culled objects pay zero per-frame cost | A | L | P2-01 | +| ENG2-P2-03 | Remove the legacy glfw/OpenGL backend (execute ADR from P0-10) | B | L | P0-10 | + +- **P2-01:** `renderer3d/core/mod.rs:54-93` — objects/instanced_meshes/lights/materials/models all `FxHashMap`; frame scan iterates hash maps. Replace with dense Vec-backed slots + generation-checked handles per RFC-0005. +- **P2-02:** Consumer evidence: `SetFrustumCullingEnabled` is ON yet Throne still needs the 80×80 terrain window because culled objects still pay BeginFrame/EndFrame cost (throne `Game3DSceneRenderer.Terrain.cs:9-14`; overhead ~41 ms regardless of zoom). Cull against the spatial index BEFORE transform/uniform/encode work. Primary fix for #678; validated by the cull_scaling bench (P0-03). +- **P2-03:** Delete the GL backend impl + glfw platform path per ADR; the `RenderBackend` trait seam stays. Frees all later work (instancing, dynamic offsets, timestamps) from dual-backend constraints. Names Throne follow-up: none (Throne is on wgpu/Metal). + +### Batch 2.2 — Upload rearchitecture (2 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-04 | Uniform v2: per-frame constants uploaded once; per-object data in dynamic-offset slots; batched buffer writes | A | L | — | +| ENG2-P2-05 | Persistent dirty-tracked per-instance transform buffers | A | M | P2-04 | +| ENG2-P2-06 | Material sort caching + retained draw list (re-sort only on scene mutation) | B | M | — | +| ENG2-P2-07 | Incremental static batching: no full re-bake on change; no CPU vertex pre-transform | B | L | — | + +- **P2-04:** `backend/wgpu_backend/uniforms.rs:78-92` — every draw command appends the full 4KB shader staging block into a ring and `write_buffer`s it (`uniforms.rs:188,211`). Split per-frame constants (camera, lights — uploaded once) from per-object data (dynamic-offset bind group slots); coalesce writes into one upload per frame. +- **P2-05:** `render_instanced.rs:28` clones `instanced_uniforms` per frame; instance buffers rebuilt wholesale. Persist per-instance buffers keyed by handle, upload only dirty ranges. +- **P2-06:** Frame scan re-sorts materials each frame; retain sorted draw list, invalidate on add/remove/material-change only. +- **P2-07:** `core_static_batch.rs:27-35` `rebuild_static_batch` re-bakes everything when `static_batch_dirty`; `core/mod.rs:159` pre-bakes *CPU-transformed vertices* into one VBO. Move to per-chunk static batches with GPU transforms; incremental add/remove. + +### Batch 2.3 — Instancing everywhere & shadows (2 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-08 | Auto-instancing by (mesh,material) identity incl. CreatePlane/CreateCube; make instancing_enabled real or remove; stop per-instance vertex clones (#679) | A | L | P2-04 | +| ENG2-P2-09 | Instanced shadow pre-pass | B | M | P2-08 | +| ENG2-P2-10 | Persistent shadow target + caster caching; kill the 300–600 ms Metal stall (#677) | B | L | P0-01 | + +- **P2-08:** `ffi/renderer3d/primitives.rs:47-176` creates one discrete object per call while `core_primitives.rs:58` `create_instanced_primitive` exists unused by that path; `renderer3d/config.rs:52` `instancing_enabled` defaults true but changes nothing for primitives. Key draw submission on (mesh id, material id) and batch automatically. Throne follow-up: THR-A-02. +- **P2-09:** Shadow pass currently draws casters individually (`shadow_pass.rs:348` per-object write_buffer). Render instanced groups instanced in the shadow pre-pass too. +- **P2-10:** Consumer note (throne `Game3DSceneRenderer.cs:221-230`): shadows disabled engine-wide due to 300–600 ms/frame stalls at 1.4k instances; root-cause with the new gpu_shadow timings (P0-01), keep the shadow RT/pipelines persistent across frames, cache the caster list with dirty invalidation. Throne follow-up: THR-A-04. + +### Batch 2.4 — Scene-scale primitives & gate validation (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-11 | Chunked/scrollable 3D tilemap-grid primitive (windowless terrain) | A | L | P2-07, P2-08 | +| ENG2-P2-12 | O(visible) validation on S1/S2; fix residual per-object frame costs; close #678 | B | M | all P2 | + +- **P2-11:** Throne destroys/recreates thousands of planes on every 12-tile recenter (`Terrain.cs:45-78`) because no grid primitive exists. One handle = one tile grid (N×M, per-tile material/UV), chunked static batches, scroll/rebind without destroy/create. Throne follow-up: THR-A-03. +- **P2-12:** Run S1/S2 + cull_scaling; attribute any remaining O(total) cost with the P0 counters; fix stragglers (per-object hashmap touches, per-frame allocs). Closing PR posts the capture report on #678. + +### Phase 2 Gate (M-series, `scripts/perf-capture`, 3-run median) +- [ ] **S1:** BeginFrame+EndFrame ≤ 4 ms combined (baseline ~42.5 ms); sustained ≥ 60 FPS. +- [ ] **S2** (100k objects, 10k visible, shadows ON): ≥ 60 FPS average; p99 frame ≤ 33 ms; no frame > 50 ms. +- [ ] **cull_scaling** (CI-gated from now on): fixed 5k visible, total 10k→100k ⇒ frame-time increase ≤ 10%. +- [ ] **Draw-call bench:** 40k CreatePlane tiles ⇒ ≤ 64 draw calls (baseline ~11k+). +- [ ] **Shadows:** S2 shadows-ON vs OFF total-frame delta ≤ 25%; `gpu_shadow` ≤ 4 ms and non-zero. +- [ ] #677, #678, #679 closed with capture links satisfying the perf definition-of-done. + +--- + +## 5. Phase 3 — Data Core v2: ECS & Store Unification (W2-store) — *requires Phase 1 gate; may overlap Phase 2 Batches 2.2+* + +**Goal:** One World-owned component store; one lock; sound FFI reads; dirty transforms. 9 issues. + +### Batch 3.1 — Decision record + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P3-01 | RFC-0006: Unified World-owned component store + locking model | — | M | Phase 1 gate | + +- Today three stores: typed World SparseSets (`ecs/world/mod.rs:118-146`), `ffi/component/storage.rs`, and `component_ops/storage.rs:265` global `CONTEXT_COMPONENT_STORAGE` — two near-identical ~400-LOC unsafe byte stores, entities in World but FFI component bytes in a disjoint global side-table. RFC decides: type-erased columnar view over World SparseSets vs World-owned byte columns; lock granularity; FFI pointer contract. + +### Batch 3.2 — Store unification (serial chain + soundness group) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P3-02 | Contiguous strided Vec\ component storage (replace Vec<*mut u8>, one malloc per component) | A | L | P3-01 | +| ENG2-P3-03 | World-owned store: delete CONTEXT_COMPONENT_STORAGE + COMPONENT_TYPE_REGISTRY globals | A | L | P3-02 | +| ENG2-P3-04 | Single context-borrow locking; batch-first internals; batch/single liveness-validation parity | A | M | P3-03 | +| ENG2-P3-05 | Fix component_get/get_mut raw-pointer-escapes-mutex soundness (copy-out or pinned arena) | B | M | P3-03 | + +- **P3-02:** `component_ops/storage.rs:38` `data: Vec<*mut u8>` with `alloc(layout)` per insert (:123-136) — 100k entities × N types = 100k+ mallocs, pointer-chasing, and `goud_component_get_all` returns scattered pointers (`ffi/component/query.rs:247-261`). Dense strided column ⇒ base pointer + stride for bulk reads. +- **P3-03:** `single_ops.rs:100-114` writes bytes into the global map keyed by `context_key` (`helpers.rs:17`) after checking liveness on the World — dual source of truth; context destroy orphans storage. Store moves into `GoudContext`/World; purge paths become structural. +- **P3-04:** 3 global Mutex acquisitions per single op (`single_ops.rs:70,92,106`, repeated at :182-323) = 120k lock ops/frame at 40k entities; batch path skips liveness entirely (`batch_ops.rs:84-101`). One borrow per op, one per batch; identical validation semantics both paths. +- **P3-05:** `ffi/component/access.rs:206` returns `*const u8` into mutex-owned memory after the guard drops — use-after-free class. Per RFC: copy-out into caller buffer, or stable-address arena + debug-enforced single-thread contract. + +### Batch 3.3 — Systems & hardening (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P3-06 | Dirty-tracked transform propagation via existing changed_ticks; iterative subtree walks | A | M | — | +| ENG2-P3-07 | Extract physics/collision/broad_phase out of ecs/; dedupe 2D/3D transform module pairs | B | L | — | +| ENG2-P3-08 | Despawn/teardown purge verification + leak canary CI (absorbs #273) | C | M | P3-03 | +| ENG2-P3-09 | FFI soundness hardening: unsafe audit (#270) + fuzz harness (#267) over the new store | C | M | P3-05 | + +- **P3-06:** `propagate3d.rs:58-128` walks EVERY archetype/entity every frame, allocating scratch Vecs, recursing subtrees (`:140-169`) — O(total) regardless of culling, despite `added_ticks/changed_ticks` existing (`sparse_set/core.rs:103-106`). Matches the BeginFrame-scales-with-total complaint from the ECS side. +- **P3-07:** ecs/ bundles collision (2906 LOC), broad_phase (1193), physics_world (1237) plus near-parallel transform/transform2d, global_transform/2d, propagate2d/3d pairs. Extract to sibling modules; unify propagation over a dimension-generic core. +- **P3-08:** The audit **refuted** the despawn-leak claim (FFI single+batch despawn purge both stores) — verify the remaining suspects only: recursive despawn and context-teardown purge of the (now World-owned) store; add churn canary (10k frames × 1k spawn/despawn, assert flat RSS) to CI. +- **P3-09:** `RawComponentStorage`'s hand-written `unsafe impl Send/Sync` (`storage.rs:51-56`) was justified by the global mutex being deleted here — re-audit all unsafe in the new store; add cargo-fuzz targets for component add/get/remove/batch and entity lifecycle FFI. + +### Phase 3 Gate +- [ ] Zero global component statics remain (`CONTEXT_COMPONENT_STORAGE`, `COMPONENT_TYPE_REGISTRY` deleted); lint/grep gate added. +- [ ] `ffi_component_benchmarks`: single component op takes exactly 1 lock (bench-asserted); `component_get_all` at 100k returns base-pointer+stride, ≥ 2x baseline throughput. +- [ ] Transform propagation: 100k entities, 1% dirty ⇒ ≤ 0.5 ms (new bench, CI-gated). +- [ ] Soak: 10k frames × 1k spawn/despawn ⇒ RSS flat ±1%; teardown leaves no orphaned storage. +- [ ] Pointer-escape fixed with regression test; fuzz targets running in nightly CI with zero findings; miri/asan lane green on component tests. + +--- + +## 6. Phase 4 — FFI v2 Convergence (W2-ffi) — *requires Phase 2 AND Phase 3 gates* + +**Goal:** The FFI stops being a per-object, per-call surface: command buffer + fn-table + bulk APIs, propagated to all 10 SDKs. 8 issues. + +### Batch 4.0 — Decision record + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-01 | RFC-0007: Command-buffer FFI + versioned fn-table entry point | — | M | P2/P3 gates | + +- 682 flat `#[no_mangle]` exports (+1347 generated JNI fns) with no versioning or bulk submission (`ffi/` audit). RFC decides opcode format, shared-memory ownership, drain semantics (one lock per frame), fn-table versioning (GDExtension-style), and the deprecation path for per-object exports (binding: no compat shims long-term). + +### Batch 4.1 — Core mechanisms (2 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-02 | Command-buffer FFI core: packed opcode buffer, engine drains under one lock per frame | A | L | P4-01 | +| ENG2-P4-03 | Versioned fn-table entry point; flat exports become generated veneer over it | B | L | P4-01 | + +- **P4-02:** Target 10k–50k ops per frame with O(1) FFI crossings. Ops: component write/read-request, transform set, spawn/despawn, draw-submit. Batch APIs (`component/batch.rs:72,145-172`) already prove the amortization pattern. +- **P4-03:** Single `goud_get_interface(version)` returning a fn-pointer struct; codegen (healthy after Phase 1) regenerates SDK bindings against it; ABI version negotiation test. + +### Batch 4.2 — Bulk surfaces (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-04 | Bulk transform-upload API: SoA handle/position/rotation arrays in one call, renderer-side dirty integration | A | M | P4-02 | +| ENG2-P4-05 | Batch-first surface: CreatePlane/CubeBatch, batched RPC drain, demote per-object hot calls | B | M | P4-02 | +| ENG2-P4-06 | Spawn/despawn churn fast path + cross-FFI object pooling | C | M | P3 gate | + +- **P4-04:** Throne calls `SetModelPosition`/`SetModelRotation` per entity per frame (throne `Entities.cs:56,287`); batch equivalents exist but were never the primary path. One call updates N instances; feeds P2-05 persistent instance buffers. Throne follow-up: THR-A-02. +- **P4-05:** SDK shape encourages the 40k-call pattern (`GoudGame.g.cs:676,682` CreateCube/CreatePlane 1:1). Add batched creates; networking drain returns all pending messages per call (consumer audit: "RPC drain 1 msg/call"); per-object exports marked deprecated in manifest metadata. +- **P4-06:** Engine-side freelist/pool for entity+component slots across FFI churn — replaces Throne's ConcurrentBag stub (throne engine-dependencies.md:15). + +### Batch 4.3 — Propagation & validation (serial) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-07 | Propagate FFI v2 through codegen to all 10 SDKs + integration tests | — | M | P4-02..06 | +| ENG2-P4-08 | S3 ffi-churn gate validation + locks/allocs-per-op CI budget benches | — | M | P4-07 | + +### Phase 4 Gate +- [ ] **S3:** the full per-frame op stream (50k reads, 10k writes, 1k spawn, 1k despawn) executes in ≤ 3 FFI crossings and ≤ 2 ms engine-side; RSS flat over 10k frames. +- [ ] fn-table entry point live; version-negotiation test passes; flat exports generated from the same manifest. +- [ ] All 10 SDKs regenerate green; parity gates pass; C#/TS/Python integration tier exercises command buffer + bulk transforms. +- [ ] locks-per-op and allocs-per-op budgets asserted in CI benches. + +### ══ SYNC A — Throne adoption (filed in throne_ge; this validates the prime directive) ══ + +| ID | Title | Effort | Blocked by | +|---|---|---|---| +| THR-A-01 | Bump GoudEngine to v2 surface; migrate breaking API changes; re-profile (supersedes GE#596; maps throne_ge#1040 and GE#590–593 content) | M | Phase 4 gate | +| THR-A-02 | Adopt batch spawn + bulk transform upload + command-buffer submission; delete per-entity SetModelPosition/SetModelRotation loops | M | THR-A-01 | +| THR-A-03 | Replace the 80×80 terrain window with the engine chunked tilemap-grid primitive (full 200×200 map) | M | THR-A-01 | +| THR-A-04 | Re-enable shadows; validate the prime-directive gate in Throne itself | S | THR-A-02, THR-A-03 | + +**Sync A acceptance (posted on both repos' tracking issues):** Throne GameWorld, full 200×200 map, no terrain window, shadows ON: ≥ 60 FPS on M-series; BeginFrame+EndFrame ≤ 4 ms; grep gate shows zero per-entity transform-push loops. + +--- + +## 7. Phase 5 — Parallelism & Determinism (W3) — *concurrent with Phases 6, 7* + +**Goal:** Multicore engine loop; determinism guarantees Throne Phase 1 requires. 6 issues. + +### Batch 5.1 — Send + schedule (serial) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P5-01 | Make World/GoudContext Send; eliminate remaining global-static blockers | — | L | Phase 4 gate | +| ENG2-P5-02 | Route the runtime loop through the existing ParallelSystemStage | — | M | P5-01 | + +- **P5-01:** `context.rs:22` documents not-Send; all contexts behind one global Mutex (`registry.rs:346`); Phase 3 removed the component-store blockers — finish the job (thread-local window/immediate state maps at `window/state.rs:361`, `immediate.rs:367`). +- **P5-02:** A working parallel scheduler with access-conflict analysis exists (`schedule/parallel/execution.rs:85` rayon::scope) but `App::run_once` drives sequential `SystemStage` (`ecs/app/mod.rs:163-165`). Wire it; feature-flag for fallback during rollout. + +### Batch 5.2 — Jobs, determinism, pacing (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P5-03 | Work-stealing job system for engine-internal work (propagation, culling, batch bake) | A | L | P5-01 | +| ENG2-P5-04 | Deterministic iteration order + world-state hash FFI hook | B | M | — | +| ENG2-P5-05 | Frame pacing: target-FPS limiter without vsync + fixed-timestep render interpolation | C | M | — | + +- **P5-03:** Replaces Throne's `Parallel.For` stub (throne engine-dependencies.md:17); deterministic scheduling required (Throne phase-1.md:229). +- **P5-04:** Throne Phase 1 gate needs same-seed identical state after 10k ticks across thread schedules (`phases/phase-1.md:224-229`); expose `goud_world_state_hash` so the consumer's determinism test can compare runs. +- **P5-05:** No frame limiter exists (pacing is wgpu present_mode only, `init.rs:73-76`); `physics_world/interpolation.rs` is a 26-LOC stub despite the real accumulator loop in `sdk/game/instance_fixed_timestep.rs:37-48`. + +### Batch 5.3 — Scale validation (serial) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P5-06 | H1 headless validation: 100k/1M published tick times, spawn/despawn soak, determinism CI job | — | M | P5-02..04 | + +### Phase 5 Gate +- [ ] **H1:** 100k entities headless tick ≤ 5 ms; 1M ≤ 33 ms avg / 50 ms max; report committed. +- [ ] Determinism: 3 seeds × 10k ticks ⇒ identical world hash, parallel stage ON, CI-gated. +- [ ] Parallel propagation+culling ≥ 2.5× speedup on 8 cores vs single thread (bench). +- [ ] Frame limiter holds 60 ± 1 FPS on S1 with vsync off. + +**══ SYNC B (throne_ge):** `THR-B-01` [M] — replace the Parallel.For/job stubs with the engine job system; wire `goud_world_state_hash` into `DeterminismTests`; re-run Throne Phase 1 scale gates against engine primitives. + +--- + +## 8. Phase 6 — 2D Render v2 (W4) — *concurrent with Phases 5, 7* + +**Goal:** Four sprite renderers become one instanced core; 2D camera and culling move engine-side. 8 issues. + +### Batch 6.0 — Decision record + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-01 | RFC-0008: Unified instanced 2D sprite core | — | S | Phase 4 gate | + +- Four parallel renderers today: immediate FFI (~15 GL-state calls/sprite, `ffi/renderer/draw/internal.rs:107-132`), batched FFI (full rebuild/sort/upload per call, `draw/batch.rs:171-343`), retained ECS SpriteBatch (unreachable from FFI, `rendering/sprite_batch/batch.rs`), WASM batcher (`wasm/sprite_renderer/renderer_core.rs`). Zero GPU instancing anywhere — every sprite CPU-expanded to 4 vertices. + +### Batch 6.1 — Core + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-02 | Instanced sprite core: static unit quad + dirty-tracked per-instance buffer (incl. flip flags, #424) | — | L | P6-01 | + +- Per-instance {transform, size, uv-rect, color, tex-index, flags}; persistent buffers, changed-only uploads (replaces `batch.rs:234-242` CPU corner rotation + `batch.rs:355-417` per-frame vertex gen). Reuses Phase 2 instancing patterns behind the backend trait. + +### Batch 6.2 — Path unification (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-03 | Route FFI sprite paths through the core; delete immediate + rebuild-per-call paths | A | M | P6-02 | +| ENG2-P6-04 | Unify retained ECS SpriteBatch + WASM batcher onto the core; delete duplicates | B | M | P6-02 | +| ENG2-P6-05 | First-class 2D camera (view+proj in shader) + engine-side 2D culling + coordinate-origin option (#554) | C | M | — | + +- **P6-05:** FFI batch sets only `u_viewport` (`draw/batch.rs:355`); Throne does world→screen math + visibility in C# per sprite (`GoudRenderPort.cs:275`). Camera uniform + grid/quadtree culling before submission; #554's configurable origin lands in the camera. + +### Batch 6.3 — 2D scale features (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-06 | Chunked tilemap renderer from TileLayer gids | A | M | P6-02 | +| ENG2-P6-07 | Auto atlas/texture-array integration with the sprite core | B | M | P6-02 | +| ENG2-P6-08 | Shaped-text run cache + S4 sprite-storm gate validation | C | M | P6-03 | + +- **P6-06:** `assets/loaders/tiled_map/layer.rs:22` TileLayer is raw `Vec` gids with no renderer — the direct cause of per-tile DrawSprite in 2D consumers. +- **P6-07:** ShelfPacker + full atlas FFI exist (`rendering/texture_atlas/packer.rs`, `ffi/renderer/atlas/ffi.rs`) but nothing auto-packs; collapse many textures into few draws without consumer bookkeeping. +- **P6-08:** `ffi/renderer/text/draw_impl.rs:45` re-shapes every string every frame (atlas is cached, shaping is not); cache runs keyed (string, font, size). + +### Phase 6 Gate +- [ ] **S4:** 100k sprites / 10k visible ≥ 60 FPS; ≤ 20 draw calls; CPU submit ≤ 2 ms. +- [ ] Exactly one sprite renderer remains (lint gate: immediate path + duplicate batchers deleted; wasm routes through the core). +- [ ] 2D example runs with zero consumer-side world→screen math. +- [ ] 200×200 tile layer renders in ≤ 4 draw calls. + +--- + +## 9. Phase 7 — Runtime Services (W5) — *concurrent with Phases 5, 6* + +**Goal:** The built-but-unwired services become real: async assets, mipmaps, one audio stack, one 2D physics, retained UI, multi-window. 6 issues. + +### Batch 7.1 — Assets & media (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P7-01 | Wire async asset pipeline into the runtime; background decode incl. RGB→RGBA off main thread | A | M | Phase 4 gate | +| ENG2-P7-02 | Mipmap generation + trilinear sampling in the wgpu texture path | B | M | — | +| ENG2-P7-03 | Unify the dual rodio audio stacks onto the provider architecture | C | M | — | + +- **P7-01:** `assets/server/async_operations.rs:44-104` (load_async + process_loads) has ZERO non-test native callers; the live path is blocking `operations.rs:14-53` incl. per-pixel RGB→RGBA on the main thread (`texture.rs:59-70`) — violates assets/CLAUDE.md's own rule. Call `process_loads()` once per frame in the instance runtime; decode on rayon. +- **P7-02:** `mip_level_count: 1` everywhere (`texture.rs:41`, `init.rs:160`, sdl/xbox init) with `MipmapFilterMode::Nearest` and no chain — aliasing + wasted bandwidth at distance. +- **P7-03:** `assets/audio_manager/` (2832 LOC) vs `libs/providers/impls/rodio_audio.rs:118-138` — two full rodio wrappers, two unsafe Send/Sync blocks; keep the provider-trait one, bridge asset data through it. + +### Batch 7.2 — Physics, UI, windowing (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P7-04 | Consolidate 2D physics on rapier2d; delete the custom ECS solver | A | M | P3-07 | +| ENG2-P7-05 | UI v2: retained command buffer, dirty-subtree relayout, string interning; absolute positioning (#553) | B | L | — | +| ENG2-P7-06 | Multi-window support (WindowId-keyed window collection) | C | M | — | + +- **P7-04:** Custom solver (`ecs/physics_world/simulation.rs`, 397 LOC) coexists with the rapier2d provider — divergence risk; rapier2d becomes canonical ("no legacy code kept"). +- **P7-05:** `manager/render.rs:12-20` rebuilds the full command Vec + clones label/font/path strings per frame; single global `layout_dirty` flag relayouts the whole tree (`layout.rs:16-33`); HashMap node storage (`manager.rs:44`). Retain commands, dirty subtrees, interned strings/slotmap; ship #553's absolute-positioning mode as part of the layout rework. +- **P7-06:** `winit_platform.rs:37-48` holds one `Option>`; needed for Phase 9 editor/tools. + +### Phase 7 Gate +- [ ] Load burst (100 × 2048² textures) during S1 ⇒ no frame > 20 ms. +- [ ] Textures upload with full mip chains (mip_level_count > 1 asserted in test); trilinear default. +- [ ] Exactly one audio stack and one 2D physics backend in-tree. +- [ ] UI: 1k-widget tree, single label change ⇒ relayout+rebuild ≤ 0.3 ms (bench). +- [ ] Two windows render simultaneously in an example. + +--- + +## 10. Phase 8 — Capability Gaps (W7) — *requires Phases 5+6+7* + +**Goal:** "Any game" credibility: navigation, particles, save/load, procedural primitives, terrain. Every capability ships FFI + 10-SDK parity + integration test + docs page (per the Phase 1 gates, now cheap). 11 issues. + +### Batch 8.1 — Highest-demand capabilities (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P8-01 | Grid A* + flow fields reusing ecs/spatial_grid; nav FFI surface (#546) | A | L | P5-03 | +| ENG2-P8-02 | Particles: FFI exposure of the existing 3D emitter + new 2D emitter on the sprite core | B | M | P6-02 | +| ENG2-P8-03 | World snapshot save/load (serde over the component registry) | C | L | P3 gate | +| ENG2-P8-04 | Seedable RNG + noise crate with FFI exports (#548) | D | S | — | + +- **P8-01:** No navigation module exists anywhere (grep-verified; confirms #546); reuse `ecs/spatial_grid/core.rs`; flow fields align with Throne's hierarchical-pathfinding plan (throne phase-1.md:138-143). +- **P8-02:** `ParticleEmitter` fully implemented (`libs/graphics/renderer3d/types.rs:107-278`, rendered in `render_instanced.rs:13`) but `ffi/renderer3d/` has no particles module and zero SDK surface. +- **P8-03:** Only network-delta + one-way scene JSON exist (`core/serialization/`, `ffi/scene_loading.rs:47`); reuse the delta code's component walk. +- **P8-04:** No `rand`/`noise` in engine deps; cross-language determinism requires an engine-owned seeded PRNG (ties to P5-04 hashing). + +### Batch 8.2 — Gameplay services (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P8-05 | Engine event bus with FFI subscription (#547) | A | M | — | +| ENG2-P8-06 | Isometric projection/tile utils (#550) | B | S | P6-05 | +| ENG2-P8-07 | 3D animation blend trees + 2-bone IK (extend the 2D controller pattern) | C | L | — | +| ENG2-P8-08 | cgmath → core/math migration (deprecated dep, 36 files) | D | M | — | + +- **P8-07:** 2D has a real state machine with crossfade (`ecs/systems/animation_controller/system.rs:74-192`); 3D is raw clip playback only, zero IK hits repo-wide. +- **P8-08:** Deliberately late: Phases 2/3/6 already deleted much cgmath-using code; migrate the remainder onto `core/math`, drop the advisory-ignored dep. + +### Batch 8.3 — Big-world (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P8-09 | Navmesh generation/query for 3D navigation | A | L | P8-01 | +| ENG2-P8-10 | Chunked heightmap terrain component | B | L | P2-11 | +| ENG2-P8-11 | Terrain LOD + texture/mesh streaming | B | L | P8-10, P7-01 | + +### Phase 8 Gate +- [ ] Nav bench: 10k agents repathing within ≤ 5 ms/frame budget; A*-parity correctness tests. +- [ ] Every new capability: FFI exports + all-SDK parity gate green + integration test + mdBook page (checklist per issue). +- [ ] Save/load: S1-scale world round-trips to identical world hash (P5-04 hook). +- [ ] cgmath absent from goud_engine Cargo.toml. + +**══ SYNC C (throne_ge):** `THR-C-01` [S] — evaluate engine A*/flow-field, event bus, and RNG/noise against Throne's C# implementations; adopt where they beat the stubs; report findings on the engine-dependencies table. + +--- + +## 11. Phase 9 — Authoring, Platforms, Examples & Polish (W8 + W6-platform) + +**Goal:** Every kept platform is CI-real; developers can see and author scenes; examples prove the v2 API. 9 issues. + +### Batch 9.1 — Platforms & tooling (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P9-01 | Scene inspector MVP (egui or web client over the MCP relay): entity tree + live component editing | A | L | P7-06 | +| ENG2-P9-02 | Inspector: scene/prefab authoring round-trip to the JSON scene format | A | M | P9-01 | +| ENG2-P9-03 | Mobile CI real: Android NDK cross-build + on-device smoke, iOS build job (#134) | B | L | Phase 1 gate | +| ENG2-P9-04 | Console targets: explicit experimental feature-gate, dedupe 3×323-line init copies, strategy record (#135) | C | M | — | +| ENG2-P9-05 | Networking truth: rename fake WebRTC to udp_datachannel or adopt a real crate; networking-tier decision record | D | M | — | + +- **P9-01/02:** The MCP debugger is real (20 tools incl. inspect_entity/capture_frame, `tools/goudengine-mcp/src/server/tools.rs`) but there is no authoring tool at all — scenes are code/JSON only (`ffi/scene_loading.rs:47`; prefabs at `context_registry/scene/prefab.rs`). Build inspector on the relay; then save-back to scene JSON. +- **P9-03:** CI has no aarch64-linux-android build; jniLibs contain only .gitkeep; JNI is validated only on desktop JVM (`Cargo.toml:172` jni_smoke). Keep-all-targets decision ⇒ make it gated. +- **P9-04:** `switch_vulkan_platform.rs:52-57`/`xbox_gdk_platform.rs:55-56` are honest PoC stubs; the three wgpu init files are byte-near-identical. Extract shared init, gate behind `experimental-console`, CI-typecheck lane. +- **P9-05:** `providers/impls/webrtc_network/mod.rs:1-17` is custom UDP+STUN labeled WebRTC, `net-webrtc=[]` pulls no crate; ~7-8k LOC networking has zero production consumers — decision record on tiering (default-feature status) + honest naming. + +### Batch 9.2 — Examples, docs, agent configs (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P9-06 | Compiled Rust examples in the engine crate (currently zero .rs examples) | A | M | — | +| ENG2-P9-07 | Example games refresh: 2–3 games validating the v2 API (consolidates #333/#336/#340/#342/#345/#348/#139) | B | L | P9-06 | +| ENG2-P9-08 | Agents/skills v2 update: perf-capture skill, render-v2 patterns, perf-work rules in .agents/ | C | M | — | +| ENG2-P9-09 | mdBook v2 architecture rewrite + breaking-changes migration guide | D | M | — | + +### Phase 9 Gate +- [ ] Inspector connects to a running S1, edits a transform live, saves scene JSON that reloads identically. +- [ ] Android + iOS CI jobs required; console typecheck lane explicit experimental; networking naming/tiering resolved. +- [ ] ≥ 2 example games + Rust examples build in required CI against the v2 API. +- [ ] Migration guide published; .agents/ + skills reference v2 patterns and the perf-capture procedure. + +--- + +## 12. Existing open-issue disposition map (requirement 7) + +| Issue | Disposition | +|---|---| +| #704 security deps | → ENG2-P1-10 | +| #679 CreatePlane bypasses instancing | → ENG2-P2-08 (closed at Phase 2 gate) | +| #678 frame cost scales with total | → ENG2-P2-02 + ENG2-P2-12 (closed at Phase 2 gate) | +| #677 shadow Metal stall | → ENG2-P2-10 (closed at Phase 2 gate) | +| #670 NuGet targets | → verify+path-fix+close in ENG2-P0-12; consumer removal in THR-S0-01 | +| #657 jni 0.21→0.22 | → ENG2-P1-02 | +| #596 V2-P08 Throne bump | close, superseded by THR-A-01 | +| #593/#592/#591/#590 target:throne | migrate to throne_ge (THR-A-01..04), close GE-side in ENG2-P0-12 | +| #554 coordinate origin | → ENG2-P6-05 | +| #553 UiManager absolute positioning | → ENG2-P7-05 | +| #550 isometric utils | → ENG2-P8-06 | +| #548 noise | → ENG2-P8-04 | +| #547 event bus | → ENG2-P8-05 | +| #546 A* | → ENG2-P8-01 | +| #475 ALPHA-002 master / #114 ALPHA-001 | close, superseded by the new v2 master tracking issue (ENG2-P0-12) | +| #425 wasm-pack deprecation | → ENG2-P1-05 | +| #424 flipX/flipY | → ENG2-P6-02 | +| #423 drawText web | close as invalid after verify (exists at `wasm/rendering.rs:239`) — ENG2-P0-12 | +| #348/#345/#342/#340/#336/#333/#139 example games | → ENG2-P9-07 | +| #273 leak detection | → ENG2-P3-08 | +| #270 unsafe audit / #267 FFI fuzz | → ENG2-P3-09 | +| #256 perf regression detection | → ENG2-P0-05 | +| #137 F22 testing | → ENG2-P1-06 | +| #136 F21 debugger | close (shipped); authoring successor ENG2-P9-01 | +| #135 console strategy | → ENG2-P9-04 | +| #134 mobile | → ENG2-P9-03 | +| Milestones 1–9 | close as superseded after migration (ENG2-P0-12) | + +## 13. Issue count summary + +| Phase | Engine issues | Throne issues | +|---|---|---| +| P0 Instrumentation & Truth | 12 | 1 (Sync 0) | +| P1 SDK Source of Truth | 10 | — | +| P2 Render Core v2 | 12 | — | +| P3 Data Core v2 | 9 | — | +| P4 FFI v2 Convergence | 8 | 4 (Sync A) | +| P5 Parallelism & Determinism | 6 | 1 (Sync B) | +| P6 2D Render v2 | 8 | — | +| P7 Runtime Services | 6 | — | +| P8 Capability Gaps | 11 | 1 (Sync C) | +| P9 Authoring/Platforms/Polish | 9 | — | +| **Total** | **91** | **7** → **98 overall** | + +Effort mix: 9 S / 47 M / 35 L / 0 XL engine-side — every issue is sized for a single `/gh-issue --worktree` agent run producing one focused PR. Each phase doc in `docs/runbook/v2/phases/` will list the run commands per batch (issue numbers filled at creation), e.g. `Batch 2.3 Group A: /gh-issue --worktree`. + +### Critical Files for Implementation +- /Users/aramhammoudeh/dev/game/GoudEngine/goud_engine/src/libs/graphics/renderer3d/core/mod.rs (scene storage — heart of Phase 2) +- /Users/aramhammoudeh/dev/game/GoudEngine/goud_engine/src/libs/graphics/backend/wgpu_backend/frame.rs (phase timing + frame path — Phase 0/2) +- /Users/aramhammoudeh/dev/game/GoudEngine/goud_engine/src/component_ops/storage.rs (global component store to be deleted — Phase 3) +- /Users/aramhammoudeh/dev/game/GoudEngine/codegen/validate_coverage.py (source-of-truth/parity gates — Phase 1) +- /Users/aramhammoudeh/dev/game/throne_ge/docs/runbook/phase-index.md (runbook convention to mirror) diff --git a/docs/src/runbook/phases/phase-0.md b/docs/src/runbook/phases/phase-0.md new file mode 100644 index 00000000..6aaed64d --- /dev/null +++ b/docs/src/runbook/phases/phase-0.md @@ -0,0 +1,65 @@ +# Phase 0 — Instrumentation, Truth & Test Foundation + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +> **Expansion:** Phase 0 adds the testing/infra foundation (Docker CI, story gallery, alloc-budget gate, spec-test convention, E2E harness, determinism/fuzz fixtures, CI restructure) — see [phase-specs-expansion.md](../phase-specs-expansion.md) §C. Total P0 issues = 19. + +--- + +## Phase 0 — Instrumentation & Truth (W0) + +**Goal:** Measure before optimizing. Every later perf claim must be attributable to a non-zero phase counter and gated by a bench. Decision records for the two big redesigns are written here with fresh data. 12 issues. + +### Batch 0.1 — Instrumentation core (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P0-01 | Add GPU timestamp queries (wgpu QuerySet) for shadow/render/submit phases | A | M | — | +| ENG2-P0-02 | Fix always-zero CPU phase counters and surface both timing families via FFI + MCP | A | M | P0-01 | +| ENG2-P0-03 | Bench suite expansion: cull_scaling, primitive draw-call, real-GPU shadow benches | B | L | — | +| ENG2-P0-04 | Feature-gated tracy/puffin profiler integration | C | M | — | + +- **P0-01:** `timestamp_writes: None` at `backend/wgpu_backend/frame.rs:148` and `shadow_pass.rs:76,382` — no GPU timing exists; all phases are `Instant::now()` CPU record-time (`frame.rs:18-127`). Add QuerySet begin/end around shadow, main pass, submit; new `gpu_shadow/gpu_render/gpu_submit` phases distinct from CPU-record phases. This is why shadow_pass reports 0 µs while Metal stalls 300–600 ms (#677 evidence). +- **P0-02:** Consumer profile shows `uniform_upload, render_pass, gpu_submit, surface_present, shadow_build, bone_*` all 0 µs (throne profile :44-52). Wire real values through `frame_timing.rs` → `GetFramePhaseTimings` FFI → MCP `get_metrics_trace` (tools/goudengine-mcp inherits the mis-attribution today). +- **P0-03:** `benches/helpers/scene3d.rs:81` force-disables frustum culling in every bench scene; `renderer3d_frame_benchmarks.rs:29,35` scale only total count — bug #678 is structurally unmeasurable. Add: (a) `cull_scaling` group — fixed 5k visible, total 10k→100k, culling ON; (b) draw-call bench for `CreatePlane`/`CreateCube` (count-pinning test `tests/renderer3d_frame_counts.rs` already documents draw_calls == n); (c) opt-in real-wgpu shadow bench (NullBackend at `helpers/null_backend.rs:46-53` no-ops everything, so `shadow_record/casters_*` measure the wrong thing). +- **P0-04:** No tracy/puffin in any Cargo.toml. Feature-gate `profiling` crate spans in the frame path + ECS schedule so devs get flamegraphs beyond the 14 fixed counters. + +### Batch 0.2 — CI wiring & scene harness (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P0-05 | Wire bench-gate.py into required CI; fix benchmarks.yml bench set; retire bench-gate.sh (absorbs #256) | A | M | P0-03 | +| ENG2-P0-06 | Promote curated GPU tests to a blocking lavapipe lane | B | M | — | +| ENG2-P0-07 | Scripted consumer-scale scene harness: S1/S2/S3/S4/H1 + `scripts/perf-capture` + committed S1 baseline report | C | L | P0-01, P0-02 | +| ENG2-P0-08 | Move community-stats bot off protected main; revoke admin push token | D | S | — | + +- **P0-05:** `scripts/bench-gate.py` (ratio-normalized, baselines at `benches/baselines/criterion_baseline.json`) has ZERO references in `.github/`/`.husky/`/`verify.sh`; `benchmarks.yml:35` runs the wrong bench set, weekly-cron only, comparing nothing. Make it a required PR check over engine_tick + renderer3d_frame + the new P0-03 benches; delete redundant `bench-gate.sh`. +- **P0-06:** 96 `#[ignore]` GPU tests run only in `headless-gpu.yml` with `continue-on-error: true` (line 25) — renderer is untested in gating CI (`ci.yml:174-178` runs only `null`-named tests). Promote a curated subset to required. +- **P0-07:** Builds the five named scenes (table above) as reproducible harness scenarios (headless + capture via MCP `capture_frame`/`get_metrics_trace`), documents the 3-run-median M-series procedure, and commits the S1 baseline numbers as the regression anchor for the Phase 2 gate. +- **P0-08:** `community-stats.yml:5-6,20-25,40-41` — daily cron force-pushes `[skip ci]` commits to protected main with an admin token (122 commits pollute blame/bisect for perf work). Move to a data branch/Pages; drop the token. + +### Batch 0.3 — Decision records & repo truth (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P0-09 | RFC-0005: Render submission redesign (spatial index, dense storage, instancing, uniform strategy) | A | M | P0-07 | +| ENG2-P0-10 | ADR: Remove legacy glfw/OpenGL backend; verify live backend; reconcile doc contradiction | A | S | — | +| ENG2-P0-11 | Docs truth pass: archive ALPHA_ROADMAP.md, promote performance-roadmap.md, fix ARCHITECTURE.md, adopt perf definition-of-done | B | M | — | +| ENG2-P0-12 | Issue/label/milestone hygiene + verify-close pass (#423, #670, #136, #114, #475; migrate #590–593) | C | M | — | + +- **P0-09:** No RFC covers the redesign #677/#678/#679 require (docs/rfcs has only RFC-0001..0004). Must decide: grid vs BVH; dense-slot storage replacing `FxHashMap` (`renderer3d/core/mod.rs:54-93`); dynamic-offset uniform slots; instancing-by-identity keying. Written with fresh P0-07 baseline numbers. Blocks Phase 2 Batch 2.1. +- **P0-10:** ARCHITECTURE.md says wgpu default; ALPHA_ROADMAP.md:1246-1248 says wgpu incomplete/GL bypasses abstraction. User decision: "no legacy code kept" — record removal of the legacy-glfw-opengl path as an ADR (execution is ENG2-P2-03). Confirm the active Metal path in code first. +- **P0-11:** ALPHA_ROADMAP.md:9 falsely claims "Phase 0 (v2) Core Performance — COMPLETE" while #677/#678/#679 are open. Archive it; promote `docs/src/development/performance-roadmap.md` (current, code-verified); write the perf definition-of-done ("closed only when the named scene hits the number AND the phase counter attributes the cost") into CLAUDE.md/agents + PR template. +- **P0-12:** Dedupe legacy labels (bug/enhancement/…) into `type:*`; add `phase:v2-p*`; close milestones 1–9 as superseded; retag #677-679 `area:performance`; verify-close: #423 (drawText EXISTS in wasm — `wasm/rendering.rs:239`, close invalid), #670 (`sdks/csharp/build/GoudEngine.targets` exists and packs — fix its repo-root path-preference fallback, then close), #136 (debugger substantially shipped — close, successor is P9-01), #114/#475 (superseded by new master issue); migrate #590–593 bodies to throne_ge and close GE-side. + +### Phase 0 Gate +- [ ] S1 capture: `gpu_shadow`, `gpu_render`, `gpu_submit` report non-zero; CPU counters `uniform_upload/render_pass/surface_present` non-zero; MCP `get_metrics_trace` shows both families. +- [ ] bench-gate.py is a required PR check over the tracked set (engine_tick, renderer3d_frame, cull_scaling, draw-call, ffi_component); baselines committed. +- [ ] cull_scaling baseline recorded and documents the current O(total) defect (expected-fail documented, becomes gated in Phase 2). +- [ ] Blocking lavapipe lane required on PRs; curated GPU tests pass. +- [ ] S1 baseline report committed (re-measured post-#670-fix numbers). +- [ ] RFC-0005 and the OpenGL-removal ADR merged; ALPHA_ROADMAP.md archived; zombie milestones closed. + +**Sync 0 (throne_ge, after gate):** `THR-S0-01` [S] — bump GoudEngine, delete the `CopyGoudEngineNativeLib` workaround (`Throne.App.csproj:33-49`), confirm the ~60x load-path overhead is gone, re-profile GameWorld with the fixed phase counters and post the report (this becomes Phase 2's consumer-side "before" evidence). Maps throne_ge #1040 partially. + +--- diff --git a/docs/src/runbook/phases/phase-1.md b/docs/src/runbook/phases/phase-1.md new file mode 100644 index 00000000..8cee1fc3 --- /dev/null +++ b/docs/src/runbook/phases/phase-1.md @@ -0,0 +1,57 @@ +# Phase 1 — SDK Single Source of Truth + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +--- + +## Phase 1 — SDK Single Source of Truth (W6-core) — *concurrent with Phase 2* + +**Goal:** Parity across 10 SDKs becomes cheap (one generated source of truth) and real (CI-gated per SDK, integration-tested). Must gate before Phase 3 so FFI rewrites propagate cleanly. 10 issues. + +### Batch 1.1 — Fix the broken generators (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P1-01 | Fix JNI codegen dual-file bug; add CI assert compiled-JNI-symbols == manifest | A | L | — | +| ENG2-P1-02 | Upgrade jni crate 0.21 → 0.22 (#657) | B | S | — | + +- **P1-01:** `jni/mod.rs:10-11` compiles ONLY `generated.g.rs` (421 symbols, frozen 2026-04-02) while `codegen/gen_jni.py:21,2275` writes orphaned `generated.rs` (588 symbols) — Kotlin/Android ships missing 167 methods incl. all frame-timing FFI. Point generator at the compiled file, delete the orphan, fix `scripts/check-rs-line-limit.sh:25` + `tools/lint_layers.rs:286` which reference the dead file, and add the CI symbol-parity assert. +- **P1-02:** Straight dependency upgrade; do after P1-01 so regen is trustworthy. + +### Batch 1.2 — Single source of truth (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P1-03 | Generate ffi_mapping.json from the Rust manifest; delete the 30-entry `_TYPE_ALIASES` drift table | A | M | — | +| ENG2-P1-04 | Per-SDK export-parity CI gate: diff each generated SDK against the manifest | A | M | P1-03 | +| ENG2-P1-05 | Bring wasm/web bridge under manifest codegen + coverage gate; wasm-pack modernization (#425) | B | L | P1-03 | + +- **P1-03:** Today three sources: auto-extracted `ffi_manifest.json` (669 fns) vs hand-maintained `ffi_mapping.json` + 430KB `goud_sdk.schema.json`; `validate_coverage.py` papers over drift with a hardcoded 30-entry alias map. Normalize types at extraction, make the manifest the sole root. +- **P1-04:** `validate_coverage.py` never inspects generated SDK outputs — 671 C# DllImports vs 669 manifest fns is ungated. Promote the `sdk-parity-check` skill concept into `codegen.sh` + required CI for all 10 SDKs. +- **P1-05:** `src/wasm/*.rs` (3.9k LOC) is hand-written, 62/682 exports covered; `gen_ts_web.py` already exists. Generate the wasm bridge from the manifest, enforce coverage (explicit exclusion list for GPU-inapplicable exports), and replace deprecated wasm-pack tooling (#425). + +### Batch 1.3 — SDK reality & release cost (5 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P1-06 | Engine-integration test tier: C#/TS/Python E2E against headless engine (absorbs #137) | A | L | — | +| ENG2-P1-07 | Decouple 9-registry publishing from patch bumps; per-ecosystem dispatch | B | M | — | +| ENG2-P1-08 | Make the Rust SDK real (replace 10-line published stub) | C | M | — | +| ENG2-P1-09 | Decision + execute: #[goud_api] macros as generator of no_mangle surface, or archive | D | M | P1-03 | +| ENG2-P1-10 | Dependency health: #704 security deps + RUSTSEC ignored-advisory burndown + dep dedup | E | M | — | + +- **P1-06:** Secondary SDK tests are scaffold-only (Swift EnumTests 36 LOC; Kotlin type tests) — never touch a running engine. Add an integration tier (headless engine, spawn/query/draw-count assertions) for C#/TS/Python minimum; other SDKs keep scaffolds + parity gate. +- **P1-07:** `release.yml` is 801 lines publishing to 9 registries on every patch (version 0.0.841); `release-please-config.json` pins 20 files. Publish only on explicit tag/dispatch per ecosystem. +- **P1-08:** `sdks/rust` is a 10-line lib.rs published to crates.io. Wire it to the real native `sdk/` module surface with tests, or re-export the engine crate SDK — no stub publishing. +- **P1-09:** `goud_engine_macros` `#[goud_api]` is a third codegen mechanism across 21 files. Decision record + execution: either it *generates* the `#[no_mangle]` surface feeding the manifest (reducing hand-written FFI), or it's archived. Cannot stay ambiguous into Phase 4's fn-table work. +- **P1-10:** 8 ignored RUSTSEC advisories in deny.toml incl. anyhow UB; #704's dependency list; duplicate crate versions. + +### Phase 1 Gate +- [ ] Clean-room `./codegen.sh` green; compiled JNI symbol set == manifest (drift = 0, was 167). +- [ ] Per-SDK parity gate required in CI for all 10 SDKs; C# DllImport count == manifest count. +- [ ] wasm/web coverage == manifest minus documented exclusions, CI-gated (was 62/682). +- [ ] C#/TS/Python integration tests run a headless engine in required CI. +- [ ] A patch version bump publishes zero registries without explicit dispatch. +- [ ] RUSTSEC ignore list ≤ 2 entries, each with a written justification. + +--- diff --git a/docs/src/runbook/phases/phase-2.md b/docs/src/runbook/phases/phase-2.md new file mode 100644 index 00000000..828130f9 --- /dev/null +++ b/docs/src/runbook/phases/phase-2.md @@ -0,0 +1,69 @@ +# Phase 2 — Render Core v2 + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +> **Expansion:** Phase 2 adds ENG2-P2-13 (SoA transform store, zero-copy extract) and ENG2-P2-14 (GPU-driven culling / indirect), and raises the gate to super-high-FPS numbers — see [phase-specs-expansion.md](../phase-specs-expansion.md) §A/§B. Total P2 issues = 14. + +--- + +## Phase 2 — Render Core v2 (W1) — *concurrent with Phase 1* + +**Goal:** Per-frame cost O(visible); instancing by default; shadows usable. Closes #677/#678/#679. This is the highest-impact phase in the roadmap. 12 issues. + +### Batch 2.1 — Foundations from RFC-0005 (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-01 | Dense scene storage: slot arrays + generational handles replacing FxHashMap iteration | A | L | P0-09 | +| ENG2-P2-02 | Scene spatial index; move culling upstream of encode so culled objects pay zero per-frame cost | A | L | P2-01 | +| ENG2-P2-03 | Remove the legacy glfw/OpenGL backend (execute ADR from P0-10) | B | L | P0-10 | + +- **P2-01:** `renderer3d/core/mod.rs:54-93` — objects/instanced_meshes/lights/materials/models all `FxHashMap`; frame scan iterates hash maps. Replace with dense Vec-backed slots + generation-checked handles per RFC-0005. +- **P2-02:** Consumer evidence: `SetFrustumCullingEnabled` is ON yet Throne still needs the 80×80 terrain window because culled objects still pay BeginFrame/EndFrame cost (throne `Game3DSceneRenderer.Terrain.cs:9-14`; overhead ~41 ms regardless of zoom). Cull against the spatial index BEFORE transform/uniform/encode work. Primary fix for #678; validated by the cull_scaling bench (P0-03). +- **P2-03:** Delete the GL backend impl + glfw platform path per ADR; the `RenderBackend` trait seam stays. Frees all later work (instancing, dynamic offsets, timestamps) from dual-backend constraints. Names Throne follow-up: none (Throne is on wgpu/Metal). + +### Batch 2.2 — Upload rearchitecture (2 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-04 | Uniform v2: per-frame constants uploaded once; per-object data in dynamic-offset slots; batched buffer writes | A | L | — | +| ENG2-P2-05 | Persistent dirty-tracked per-instance transform buffers | A | M | P2-04 | +| ENG2-P2-06 | Material sort caching + retained draw list (re-sort only on scene mutation) | B | M | — | +| ENG2-P2-07 | Incremental static batching: no full re-bake on change; no CPU vertex pre-transform | B | L | — | + +- **P2-04:** `backend/wgpu_backend/uniforms.rs:78-92` — every draw command appends the full 4KB shader staging block into a ring and `write_buffer`s it (`uniforms.rs:188,211`). Split per-frame constants (camera, lights — uploaded once) from per-object data (dynamic-offset bind group slots); coalesce writes into one upload per frame. +- **P2-05:** `render_instanced.rs:28` clones `instanced_uniforms` per frame; instance buffers rebuilt wholesale. Persist per-instance buffers keyed by handle, upload only dirty ranges. +- **P2-06:** Frame scan re-sorts materials each frame; retain sorted draw list, invalidate on add/remove/material-change only. +- **P2-07:** `core_static_batch.rs:27-35` `rebuild_static_batch` re-bakes everything when `static_batch_dirty`; `core/mod.rs:159` pre-bakes *CPU-transformed vertices* into one VBO. Move to per-chunk static batches with GPU transforms; incremental add/remove. + +### Batch 2.3 — Instancing everywhere & shadows (2 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-08 | Auto-instancing by (mesh,material) identity incl. CreatePlane/CreateCube; make instancing_enabled real or remove; stop per-instance vertex clones (#679) | A | L | P2-04 | +| ENG2-P2-09 | Instanced shadow pre-pass | B | M | P2-08 | +| ENG2-P2-10 | Persistent shadow target + caster caching; kill the 300–600 ms Metal stall (#677) | B | L | P0-01 | + +- **P2-08:** `ffi/renderer3d/primitives.rs:47-176` creates one discrete object per call while `core_primitives.rs:58` `create_instanced_primitive` exists unused by that path; `renderer3d/config.rs:52` `instancing_enabled` defaults true but changes nothing for primitives. Key draw submission on (mesh id, material id) and batch automatically. Throne follow-up: THR-A-02. +- **P2-09:** Shadow pass currently draws casters individually (`shadow_pass.rs:348` per-object write_buffer). Render instanced groups instanced in the shadow pre-pass too. +- **P2-10:** Consumer note (throne `Game3DSceneRenderer.cs:221-230`): shadows disabled engine-wide due to 300–600 ms/frame stalls at 1.4k instances; root-cause with the new gpu_shadow timings (P0-01), keep the shadow RT/pipelines persistent across frames, cache the caster list with dirty invalidation. Throne follow-up: THR-A-04. + +### Batch 2.4 — Scene-scale primitives & gate validation (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P2-11 | Chunked/scrollable 3D tilemap-grid primitive (windowless terrain) | A | L | P2-07, P2-08 | +| ENG2-P2-12 | O(visible) validation on S1/S2; fix residual per-object frame costs; close #678 | B | M | all P2 | + +- **P2-11:** Throne destroys/recreates thousands of planes on every 12-tile recenter (`Terrain.cs:45-78`) because no grid primitive exists. One handle = one tile grid (N×M, per-tile material/UV), chunked static batches, scroll/rebind without destroy/create. Throne follow-up: THR-A-03. +- **P2-12:** Run S1/S2 + cull_scaling; attribute any remaining O(total) cost with the P0 counters; fix stragglers (per-object hashmap touches, per-frame allocs). Closing PR posts the capture report on #678. + +### Phase 2 Gate (M-series, `scripts/perf-capture`, 3-run median) +- [ ] **S1:** BeginFrame+EndFrame ≤ 4 ms combined (baseline ~42.5 ms); sustained ≥ 60 FPS. +- [ ] **S2** (100k objects, 10k visible, shadows ON): ≥ 60 FPS average; p99 frame ≤ 33 ms; no frame > 50 ms. +- [ ] **cull_scaling** (CI-gated from now on): fixed 5k visible, total 10k→100k ⇒ frame-time increase ≤ 10%. +- [ ] **Draw-call bench:** 40k CreatePlane tiles ⇒ ≤ 64 draw calls (baseline ~11k+). +- [ ] **Shadows:** S2 shadows-ON vs OFF total-frame delta ≤ 25%; `gpu_shadow` ≤ 4 ms and non-zero. +- [ ] #677, #678, #679 closed with capture links satisfying the perf definition-of-done. + +--- diff --git a/docs/src/runbook/phases/phase-3.md b/docs/src/runbook/phases/phase-3.md new file mode 100644 index 00000000..402ada5b --- /dev/null +++ b/docs/src/runbook/phases/phase-3.md @@ -0,0 +1,54 @@ +# Phase 3 — Data Core v2 (ECS + store unification) + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +--- + +## Phase 3 — Data Core v2: ECS & Store Unification (W2-store) — *requires Phase 1 gate; may overlap Phase 2 Batches 2.2+* + +**Goal:** One World-owned component store; one lock; sound FFI reads; dirty transforms. 9 issues. + +### Batch 3.1 — Decision record + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P3-01 | RFC-0006: Unified World-owned component store + locking model | — | M | Phase 1 gate | + +- Today three stores: typed World SparseSets (`ecs/world/mod.rs:118-146`), `ffi/component/storage.rs`, and `component_ops/storage.rs:265` global `CONTEXT_COMPONENT_STORAGE` — two near-identical ~400-LOC unsafe byte stores, entities in World but FFI component bytes in a disjoint global side-table. RFC decides: type-erased columnar view over World SparseSets vs World-owned byte columns; lock granularity; FFI pointer contract. + +### Batch 3.2 — Store unification (serial chain + soundness group) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P3-02 | Contiguous strided Vec\ component storage (replace Vec<*mut u8>, one malloc per component) | A | L | P3-01 | +| ENG2-P3-03 | World-owned store: delete CONTEXT_COMPONENT_STORAGE + COMPONENT_TYPE_REGISTRY globals | A | L | P3-02 | +| ENG2-P3-04 | Single context-borrow locking; batch-first internals; batch/single liveness-validation parity | A | M | P3-03 | +| ENG2-P3-05 | Fix component_get/get_mut raw-pointer-escapes-mutex soundness (copy-out or pinned arena) | B | M | P3-03 | + +- **P3-02:** `component_ops/storage.rs:38` `data: Vec<*mut u8>` with `alloc(layout)` per insert (:123-136) — 100k entities × N types = 100k+ mallocs, pointer-chasing, and `goud_component_get_all` returns scattered pointers (`ffi/component/query.rs:247-261`). Dense strided column ⇒ base pointer + stride for bulk reads. +- **P3-03:** `single_ops.rs:100-114` writes bytes into the global map keyed by `context_key` (`helpers.rs:17`) after checking liveness on the World — dual source of truth; context destroy orphans storage. Store moves into `GoudContext`/World; purge paths become structural. +- **P3-04:** 3 global Mutex acquisitions per single op (`single_ops.rs:70,92,106`, repeated at :182-323) = 120k lock ops/frame at 40k entities; batch path skips liveness entirely (`batch_ops.rs:84-101`). One borrow per op, one per batch; identical validation semantics both paths. +- **P3-05:** `ffi/component/access.rs:206` returns `*const u8` into mutex-owned memory after the guard drops — use-after-free class. Per RFC: copy-out into caller buffer, or stable-address arena + debug-enforced single-thread contract. + +### Batch 3.3 — Systems & hardening (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P3-06 | Dirty-tracked transform propagation via existing changed_ticks; iterative subtree walks | A | M | — | +| ENG2-P3-07 | Extract physics/collision/broad_phase out of ecs/; dedupe 2D/3D transform module pairs | B | L | — | +| ENG2-P3-08 | Despawn/teardown purge verification + leak canary CI (absorbs #273) | C | M | P3-03 | +| ENG2-P3-09 | FFI soundness hardening: unsafe audit (#270) + fuzz harness (#267) over the new store | C | M | P3-05 | + +- **P3-06:** `propagate3d.rs:58-128` walks EVERY archetype/entity every frame, allocating scratch Vecs, recursing subtrees (`:140-169`) — O(total) regardless of culling, despite `added_ticks/changed_ticks` existing (`sparse_set/core.rs:103-106`). Matches the BeginFrame-scales-with-total complaint from the ECS side. +- **P3-07:** ecs/ bundles collision (2906 LOC), broad_phase (1193), physics_world (1237) plus near-parallel transform/transform2d, global_transform/2d, propagate2d/3d pairs. Extract to sibling modules; unify propagation over a dimension-generic core. +- **P3-08:** The audit **refuted** the despawn-leak claim (FFI single+batch despawn purge both stores) — verify the remaining suspects only: recursive despawn and context-teardown purge of the (now World-owned) store; add churn canary (10k frames × 1k spawn/despawn, assert flat RSS) to CI. +- **P3-09:** `RawComponentStorage`'s hand-written `unsafe impl Send/Sync` (`storage.rs:51-56`) was justified by the global mutex being deleted here — re-audit all unsafe in the new store; add cargo-fuzz targets for component add/get/remove/batch and entity lifecycle FFI. + +### Phase 3 Gate +- [ ] Zero global component statics remain (`CONTEXT_COMPONENT_STORAGE`, `COMPONENT_TYPE_REGISTRY` deleted); lint/grep gate added. +- [ ] `ffi_component_benchmarks`: single component op takes exactly 1 lock (bench-asserted); `component_get_all` at 100k returns base-pointer+stride, ≥ 2x baseline throughput. +- [ ] Transform propagation: 100k entities, 1% dirty ⇒ ≤ 0.5 ms (new bench, CI-gated). +- [ ] Soak: 10k frames × 1k spawn/despawn ⇒ RSS flat ±1%; teardown leaves no orphaned storage. +- [ ] Pointer-escape fixed with regression test; fuzz targets running in nightly CI with zero findings; miri/asan lane green on component tests. + +--- diff --git a/docs/src/runbook/phases/phase-4.md b/docs/src/runbook/phases/phase-4.md new file mode 100644 index 00000000..9b619e8b --- /dev/null +++ b/docs/src/runbook/phases/phase-4.md @@ -0,0 +1,65 @@ +# Phase 4 — FFI v2 Convergence + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +--- + +## Phase 4 — FFI v2 Convergence (W2-ffi) — *requires Phase 2 AND Phase 3 gates* + +**Goal:** The FFI stops being a per-object, per-call surface: command buffer + fn-table + bulk APIs, propagated to all 10 SDKs. 8 issues. + +### Batch 4.0 — Decision record + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-01 | RFC-0007: Command-buffer FFI + versioned fn-table entry point | — | M | P2/P3 gates | + +- 682 flat `#[no_mangle]` exports (+1347 generated JNI fns) with no versioning or bulk submission (`ffi/` audit). RFC decides opcode format, shared-memory ownership, drain semantics (one lock per frame), fn-table versioning (GDExtension-style), and the deprecation path for per-object exports (binding: no compat shims long-term). + +### Batch 4.1 — Core mechanisms (2 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-02 | Command-buffer FFI core: packed opcode buffer, engine drains under one lock per frame | A | L | P4-01 | +| ENG2-P4-03 | Versioned fn-table entry point; flat exports become generated veneer over it | B | L | P4-01 | + +- **P4-02:** Target 10k–50k ops per frame with O(1) FFI crossings. Ops: component write/read-request, transform set, spawn/despawn, draw-submit. Batch APIs (`component/batch.rs:72,145-172`) already prove the amortization pattern. +- **P4-03:** Single `goud_get_interface(version)` returning a fn-pointer struct; codegen (healthy after Phase 1) regenerates SDK bindings against it; ABI version negotiation test. + +### Batch 4.2 — Bulk surfaces (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-04 | Bulk transform-upload API: SoA handle/position/rotation arrays in one call, renderer-side dirty integration | A | M | P4-02 | +| ENG2-P4-05 | Batch-first surface: CreatePlane/CubeBatch, batched RPC drain, demote per-object hot calls | B | M | P4-02 | +| ENG2-P4-06 | Spawn/despawn churn fast path + cross-FFI object pooling | C | M | P3 gate | + +- **P4-04:** Throne calls `SetModelPosition`/`SetModelRotation` per entity per frame (throne `Entities.cs:56,287`); batch equivalents exist but were never the primary path. One call updates N instances; feeds P2-05 persistent instance buffers. Throne follow-up: THR-A-02. +- **P4-05:** SDK shape encourages the 40k-call pattern (`GoudGame.g.cs:676,682` CreateCube/CreatePlane 1:1). Add batched creates; networking drain returns all pending messages per call (consumer audit: "RPC drain 1 msg/call"); per-object exports marked deprecated in manifest metadata. +- **P4-06:** Engine-side freelist/pool for entity+component slots across FFI churn — replaces Throne's ConcurrentBag stub (throne engine-dependencies.md:15). + +### Batch 4.3 — Propagation & validation (serial) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P4-07 | Propagate FFI v2 through codegen to all 10 SDKs + integration tests | — | M | P4-02..06 | +| ENG2-P4-08 | S3 ffi-churn gate validation + locks/allocs-per-op CI budget benches | — | M | P4-07 | + +### Phase 4 Gate +- [ ] **S3:** the full per-frame op stream (50k reads, 10k writes, 1k spawn, 1k despawn) executes in ≤ 3 FFI crossings and ≤ 2 ms engine-side; RSS flat over 10k frames. +- [ ] fn-table entry point live; version-negotiation test passes; flat exports generated from the same manifest. +- [ ] All 10 SDKs regenerate green; parity gates pass; C#/TS/Python integration tier exercises command buffer + bulk transforms. +- [ ] locks-per-op and allocs-per-op budgets asserted in CI benches. + +### ══ SYNC A — Throne adoption (filed in throne_ge; this validates the prime directive) ══ + +| ID | Title | Effort | Blocked by | +|---|---|---|---| +| THR-A-01 | Bump GoudEngine to v2 surface; migrate breaking API changes; re-profile (supersedes GE#596; maps throne_ge#1040 and GE#590–593 content) | M | Phase 4 gate | +| THR-A-02 | Adopt batch spawn + bulk transform upload + command-buffer submission; delete per-entity SetModelPosition/SetModelRotation loops | M | THR-A-01 | +| THR-A-03 | Replace the 80×80 terrain window with the engine chunked tilemap-grid primitive (full 200×200 map) | M | THR-A-01 | +| THR-A-04 | Re-enable shadows; validate the prime-directive gate in Throne itself | S | THR-A-02, THR-A-03 | + +**Sync A acceptance (posted on both repos' tracking issues):** Throne GameWorld, full 200×200 map, no terrain window, shadows ON: ≥ 60 FPS on M-series; BeginFrame+EndFrame ≤ 4 ms; grep gate shows zero per-entity transform-push loops. + +--- diff --git a/docs/src/runbook/phases/phase-5.md b/docs/src/runbook/phases/phase-5.md new file mode 100644 index 00000000..7d9e8257 --- /dev/null +++ b/docs/src/runbook/phases/phase-5.md @@ -0,0 +1,49 @@ +# Phase 5 — Parallelism & Determinism + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +> **Expansion:** Phase 5 adds ENG2-P5-07 (per-frame frame-arena allocator + extract/prepare/submit split) and raises the H1 gate — see [phase-specs-expansion.md](../phase-specs-expansion.md) §B. Total P5 issues = 7. + +--- + +## Phase 5 — Parallelism & Determinism (W3) — *concurrent with Phases 6, 7* + +**Goal:** Multicore engine loop; determinism guarantees Throne Phase 1 requires. 6 issues. + +### Batch 5.1 — Send + schedule (serial) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P5-01 | Make World/GoudContext Send; eliminate remaining global-static blockers | — | L | Phase 4 gate | +| ENG2-P5-02 | Route the runtime loop through the existing ParallelSystemStage | — | M | P5-01 | + +- **P5-01:** `context.rs:22` documents not-Send; all contexts behind one global Mutex (`registry.rs:346`); Phase 3 removed the component-store blockers — finish the job (thread-local window/immediate state maps at `window/state.rs:361`, `immediate.rs:367`). +- **P5-02:** A working parallel scheduler with access-conflict analysis exists (`schedule/parallel/execution.rs:85` rayon::scope) but `App::run_once` drives sequential `SystemStage` (`ecs/app/mod.rs:163-165`). Wire it; feature-flag for fallback during rollout. + +### Batch 5.2 — Jobs, determinism, pacing (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P5-03 | Work-stealing job system for engine-internal work (propagation, culling, batch bake) | A | L | P5-01 | +| ENG2-P5-04 | Deterministic iteration order + world-state hash FFI hook | B | M | — | +| ENG2-P5-05 | Frame pacing: target-FPS limiter without vsync + fixed-timestep render interpolation | C | M | — | + +- **P5-03:** Replaces Throne's `Parallel.For` stub (throne engine-dependencies.md:17); deterministic scheduling required (Throne phase-1.md:229). +- **P5-04:** Throne Phase 1 gate needs same-seed identical state after 10k ticks across thread schedules (`phases/phase-1.md:224-229`); expose `goud_world_state_hash` so the consumer's determinism test can compare runs. +- **P5-05:** No frame limiter exists (pacing is wgpu present_mode only, `init.rs:73-76`); `physics_world/interpolation.rs` is a 26-LOC stub despite the real accumulator loop in `sdk/game/instance_fixed_timestep.rs:37-48`. + +### Batch 5.3 — Scale validation (serial) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P5-06 | H1 headless validation: 100k/1M published tick times, spawn/despawn soak, determinism CI job | — | M | P5-02..04 | + +### Phase 5 Gate +- [ ] **H1:** 100k entities headless tick ≤ 5 ms; 1M ≤ 33 ms avg / 50 ms max; report committed. +- [ ] Determinism: 3 seeds × 10k ticks ⇒ identical world hash, parallel stage ON, CI-gated. +- [ ] Parallel propagation+culling ≥ 2.5× speedup on 8 cores vs single thread (bench). +- [ ] Frame limiter holds 60 ± 1 FPS on S1 with vsync off. + +**══ SYNC B (throne_ge):** `THR-B-01` [M] — replace the Parallel.For/job stubs with the engine job system; wire `goud_world_state_hash` into `DeterminismTests`; re-run Throne Phase 1 scale gates against engine primitives. + +--- diff --git a/docs/src/runbook/phases/phase-6.md b/docs/src/runbook/phases/phase-6.md new file mode 100644 index 00000000..b051eb6a --- /dev/null +++ b/docs/src/runbook/phases/phase-6.md @@ -0,0 +1,55 @@ +# Phase 6 — 2D Render v2 + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +--- + +## Phase 6 — 2D Render v2 (W4) — *concurrent with Phases 5, 7* + +**Goal:** Four sprite renderers become one instanced core; 2D camera and culling move engine-side. 8 issues. + +### Batch 6.0 — Decision record + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-01 | RFC-0008: Unified instanced 2D sprite core | — | S | Phase 4 gate | + +- Four parallel renderers today: immediate FFI (~15 GL-state calls/sprite, `ffi/renderer/draw/internal.rs:107-132`), batched FFI (full rebuild/sort/upload per call, `draw/batch.rs:171-343`), retained ECS SpriteBatch (unreachable from FFI, `rendering/sprite_batch/batch.rs`), WASM batcher (`wasm/sprite_renderer/renderer_core.rs`). Zero GPU instancing anywhere — every sprite CPU-expanded to 4 vertices. + +### Batch 6.1 — Core + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-02 | Instanced sprite core: static unit quad + dirty-tracked per-instance buffer (incl. flip flags, #424) | — | L | P6-01 | + +- Per-instance {transform, size, uv-rect, color, tex-index, flags}; persistent buffers, changed-only uploads (replaces `batch.rs:234-242` CPU corner rotation + `batch.rs:355-417` per-frame vertex gen). Reuses Phase 2 instancing patterns behind the backend trait. + +### Batch 6.2 — Path unification (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-03 | Route FFI sprite paths through the core; delete immediate + rebuild-per-call paths | A | M | P6-02 | +| ENG2-P6-04 | Unify retained ECS SpriteBatch + WASM batcher onto the core; delete duplicates | B | M | P6-02 | +| ENG2-P6-05 | First-class 2D camera (view+proj in shader) + engine-side 2D culling + coordinate-origin option (#554) | C | M | — | + +- **P6-05:** FFI batch sets only `u_viewport` (`draw/batch.rs:355`); Throne does world→screen math + visibility in C# per sprite (`GoudRenderPort.cs:275`). Camera uniform + grid/quadtree culling before submission; #554's configurable origin lands in the camera. + +### Batch 6.3 — 2D scale features (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P6-06 | Chunked tilemap renderer from TileLayer gids | A | M | P6-02 | +| ENG2-P6-07 | Auto atlas/texture-array integration with the sprite core | B | M | P6-02 | +| ENG2-P6-08 | Shaped-text run cache + S4 sprite-storm gate validation | C | M | P6-03 | + +- **P6-06:** `assets/loaders/tiled_map/layer.rs:22` TileLayer is raw `Vec` gids with no renderer — the direct cause of per-tile DrawSprite in 2D consumers. +- **P6-07:** ShelfPacker + full atlas FFI exist (`rendering/texture_atlas/packer.rs`, `ffi/renderer/atlas/ffi.rs`) but nothing auto-packs; collapse many textures into few draws without consumer bookkeeping. +- **P6-08:** `ffi/renderer/text/draw_impl.rs:45` re-shapes every string every frame (atlas is cached, shaping is not); cache runs keyed (string, font, size). + +### Phase 6 Gate +- [ ] **S4:** 100k sprites / 10k visible ≥ 60 FPS; ≤ 20 draw calls; CPU submit ≤ 2 ms. +- [ ] Exactly one sprite renderer remains (lint gate: immediate path + duplicate batchers deleted; wasm routes through the core). +- [ ] 2D example runs with zero consumer-side world→screen math. +- [ ] 200×200 tile layer renders in ≤ 4 draw calls. + +--- diff --git a/docs/src/runbook/phases/phase-7.md b/docs/src/runbook/phases/phase-7.md new file mode 100644 index 00000000..d9a51e46 --- /dev/null +++ b/docs/src/runbook/phases/phase-7.md @@ -0,0 +1,42 @@ +# Phase 7 — Runtime Services + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +--- + +## Phase 7 — Runtime Services (W5) — *concurrent with Phases 5, 6* + +**Goal:** The built-but-unwired services become real: async assets, mipmaps, one audio stack, one 2D physics, retained UI, multi-window. 6 issues. + +### Batch 7.1 — Assets & media (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P7-01 | Wire async asset pipeline into the runtime; background decode incl. RGB→RGBA off main thread | A | M | Phase 4 gate | +| ENG2-P7-02 | Mipmap generation + trilinear sampling in the wgpu texture path | B | M | — | +| ENG2-P7-03 | Unify the dual rodio audio stacks onto the provider architecture | C | M | — | + +- **P7-01:** `assets/server/async_operations.rs:44-104` (load_async + process_loads) has ZERO non-test native callers; the live path is blocking `operations.rs:14-53` incl. per-pixel RGB→RGBA on the main thread (`texture.rs:59-70`) — violates assets/CLAUDE.md's own rule. Call `process_loads()` once per frame in the instance runtime; decode on rayon. +- **P7-02:** `mip_level_count: 1` everywhere (`texture.rs:41`, `init.rs:160`, sdl/xbox init) with `MipmapFilterMode::Nearest` and no chain — aliasing + wasted bandwidth at distance. +- **P7-03:** `assets/audio_manager/` (2832 LOC) vs `libs/providers/impls/rodio_audio.rs:118-138` — two full rodio wrappers, two unsafe Send/Sync blocks; keep the provider-trait one, bridge asset data through it. + +### Batch 7.2 — Physics, UI, windowing (3 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P7-04 | Consolidate 2D physics on rapier2d; delete the custom ECS solver | A | M | P3-07 | +| ENG2-P7-05 | UI v2: retained command buffer, dirty-subtree relayout, string interning; absolute positioning (#553) | B | L | — | +| ENG2-P7-06 | Multi-window support (WindowId-keyed window collection) | C | M | — | + +- **P7-04:** Custom solver (`ecs/physics_world/simulation.rs`, 397 LOC) coexists with the rapier2d provider — divergence risk; rapier2d becomes canonical ("no legacy code kept"). +- **P7-05:** `manager/render.rs:12-20` rebuilds the full command Vec + clones label/font/path strings per frame; single global `layout_dirty` flag relayouts the whole tree (`layout.rs:16-33`); HashMap node storage (`manager.rs:44`). Retain commands, dirty subtrees, interned strings/slotmap; ship #553's absolute-positioning mode as part of the layout rework. +- **P7-06:** `winit_platform.rs:37-48` holds one `Option>`; needed for Phase 9 editor/tools. + +### Phase 7 Gate +- [ ] Load burst (100 × 2048² textures) during S1 ⇒ no frame > 20 ms. +- [ ] Textures upload with full mip chains (mip_level_count > 1 asserted in test); trilinear default. +- [ ] Exactly one audio stack and one 2D physics backend in-tree. +- [ ] UI: 1k-widget tree, single label change ⇒ relayout+rebuild ≤ 0.3 ms (bench). +- [ ] Two windows render simultaneously in an example. + +--- diff --git a/docs/src/runbook/phases/phase-8.md b/docs/src/runbook/phases/phase-8.md new file mode 100644 index 00000000..1d9afb3c --- /dev/null +++ b/docs/src/runbook/phases/phase-8.md @@ -0,0 +1,53 @@ +# Phase 8 — Capability Gaps + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +--- + +## Phase 8 — Capability Gaps (W7) — *requires Phases 5+6+7* + +**Goal:** "Any game" credibility: navigation, particles, save/load, procedural primitives, terrain. Every capability ships FFI + 10-SDK parity + integration test + docs page (per the Phase 1 gates, now cheap). 11 issues. + +### Batch 8.1 — Highest-demand capabilities (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P8-01 | Grid A* + flow fields reusing ecs/spatial_grid; nav FFI surface (#546) | A | L | P5-03 | +| ENG2-P8-02 | Particles: FFI exposure of the existing 3D emitter + new 2D emitter on the sprite core | B | M | P6-02 | +| ENG2-P8-03 | World snapshot save/load (serde over the component registry) | C | L | P3 gate | +| ENG2-P8-04 | Seedable RNG + noise crate with FFI exports (#548) | D | S | — | + +- **P8-01:** No navigation module exists anywhere (grep-verified; confirms #546); reuse `ecs/spatial_grid/core.rs`; flow fields align with Throne's hierarchical-pathfinding plan (throne phase-1.md:138-143). +- **P8-02:** `ParticleEmitter` fully implemented (`libs/graphics/renderer3d/types.rs:107-278`, rendered in `render_instanced.rs:13`) but `ffi/renderer3d/` has no particles module and zero SDK surface. +- **P8-03:** Only network-delta + one-way scene JSON exist (`core/serialization/`, `ffi/scene_loading.rs:47`); reuse the delta code's component walk. +- **P8-04:** No `rand`/`noise` in engine deps; cross-language determinism requires an engine-owned seeded PRNG (ties to P5-04 hashing). + +### Batch 8.2 — Gameplay services (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P8-05 | Engine event bus with FFI subscription (#547) | A | M | — | +| ENG2-P8-06 | Isometric projection/tile utils (#550) | B | S | P6-05 | +| ENG2-P8-07 | 3D animation blend trees + 2-bone IK (extend the 2D controller pattern) | C | L | — | +| ENG2-P8-08 | cgmath → core/math migration (deprecated dep, 36 files) | D | M | — | + +- **P8-07:** 2D has a real state machine with crossfade (`ecs/systems/animation_controller/system.rs:74-192`); 3D is raw clip playback only, zero IK hits repo-wide. +- **P8-08:** Deliberately late: Phases 2/3/6 already deleted much cgmath-using code; migrate the remainder onto `core/math`, drop the advisory-ignored dep. + +### Batch 8.3 — Big-world (2 groups) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P8-09 | Navmesh generation/query for 3D navigation | A | L | P8-01 | +| ENG2-P8-10 | Chunked heightmap terrain component | B | L | P2-11 | +| ENG2-P8-11 | Terrain LOD + texture/mesh streaming | B | L | P8-10, P7-01 | + +### Phase 8 Gate +- [ ] Nav bench: 10k agents repathing within ≤ 5 ms/frame budget; A*-parity correctness tests. +- [ ] Every new capability: FFI exports + all-SDK parity gate green + integration test + mdBook page (checklist per issue). +- [ ] Save/load: S1-scale world round-trips to identical world hash (P5-04 hook). +- [ ] cgmath absent from goud_engine Cargo.toml. + +**══ SYNC C (throne_ge):** `THR-C-01` [S] — evaluate engine A*/flow-field, event bus, and RNG/noise against Throne's C# implementations; adopt where they beat the stubs; report findings on the engine-dependencies table. + +--- diff --git a/docs/src/runbook/phases/phase-9.md b/docs/src/runbook/phases/phase-9.md new file mode 100644 index 00000000..150fa45e --- /dev/null +++ b/docs/src/runbook/phases/phase-9.md @@ -0,0 +1,100 @@ +# Phase 9 — Authoring, Platforms, Examples & Polish + +_Authoritative spec. See also the [phase index](../phase-index.md), [perf-dod](../perf-dod.md), and [testing-strategy](../testing-strategy.md). Live issue numbers: pinned master #810._ + +> **Expansion:** Phase 9 adds ENG2-P9-10 (self-hosted M-series nightly perf runner + trend dashboard) — see [phase-specs-expansion.md](../phase-specs-expansion.md) §D. Total P9 issues = 10. + +--- + +## Phase 9 — Authoring, Platforms, Examples & Polish (W8 + W6-platform) + +**Goal:** Every kept platform is CI-real; developers can see and author scenes; examples prove the v2 API. 9 issues. + +### Batch 9.1 — Platforms & tooling (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P9-01 | Scene inspector MVP (egui or web client over the MCP relay): entity tree + live component editing | A | L | P7-06 | +| ENG2-P9-02 | Inspector: scene/prefab authoring round-trip to the JSON scene format | A | M | P9-01 | +| ENG2-P9-03 | Mobile CI real: Android NDK cross-build + on-device smoke, iOS build job (#134) | B | L | Phase 1 gate | +| ENG2-P9-04 | Console targets: explicit experimental feature-gate, dedupe 3×323-line init copies, strategy record (#135) | C | M | — | +| ENG2-P9-05 | Networking truth: rename fake WebRTC to udp_datachannel or adopt a real crate; networking-tier decision record | D | M | — | + +- **P9-01/02:** The MCP debugger is real (20 tools incl. inspect_entity/capture_frame, `tools/goudengine-mcp/src/server/tools.rs`) but there is no authoring tool at all — scenes are code/JSON only (`ffi/scene_loading.rs:47`; prefabs at `context_registry/scene/prefab.rs`). Build inspector on the relay; then save-back to scene JSON. +- **P9-03:** CI has no aarch64-linux-android build; jniLibs contain only .gitkeep; JNI is validated only on desktop JVM (`Cargo.toml:172` jni_smoke). Keep-all-targets decision ⇒ make it gated. +- **P9-04:** `switch_vulkan_platform.rs:52-57`/`xbox_gdk_platform.rs:55-56` are honest PoC stubs; the three wgpu init files are byte-near-identical. Extract shared init, gate behind `experimental-console`, CI-typecheck lane. +- **P9-05:** `providers/impls/webrtc_network/mod.rs:1-17` is custom UDP+STUN labeled WebRTC, `net-webrtc=[]` pulls no crate; ~7-8k LOC networking has zero production consumers — decision record on tiering (default-feature status) + honest naming. + +### Batch 9.2 — Examples, docs, agent configs (4 groups, parallel) + +| ID | Title | Grp | Effort | Blocked by | +|---|---|---|---|---| +| ENG2-P9-06 | Compiled Rust examples in the engine crate (currently zero .rs examples) | A | M | — | +| ENG2-P9-07 | Example games refresh: 2–3 games validating the v2 API (consolidates #333/#336/#340/#342/#345/#348/#139) | B | L | P9-06 | +| ENG2-P9-08 | Agents/skills v2 update: perf-capture skill, render-v2 patterns, perf-work rules in .agents/ | C | M | — | +| ENG2-P9-09 | mdBook v2 architecture rewrite + breaking-changes migration guide | D | M | — | + +### Phase 9 Gate +- [ ] Inspector connects to a running S1, edits a transform live, saves scene JSON that reloads identically. +- [ ] Android + iOS CI jobs required; console typecheck lane explicit experimental; networking naming/tiering resolved. +- [ ] ≥ 2 example games + Rust examples build in required CI against the v2 API. +- [ ] Migration guide published; .agents/ + skills reference v2 patterns and the perf-capture procedure. + +--- + +## 12. Existing open-issue disposition map (requirement 7) + +| Issue | Disposition | +|---|---| +| #704 security deps | → ENG2-P1-10 | +| #679 CreatePlane bypasses instancing | → ENG2-P2-08 (closed at Phase 2 gate) | +| #678 frame cost scales with total | → ENG2-P2-02 + ENG2-P2-12 (closed at Phase 2 gate) | +| #677 shadow Metal stall | → ENG2-P2-10 (closed at Phase 2 gate) | +| #670 NuGet targets | → verify+path-fix+close in ENG2-P0-12; consumer removal in THR-S0-01 | +| #657 jni 0.21→0.22 | → ENG2-P1-02 | +| #596 V2-P08 Throne bump | close, superseded by THR-A-01 | +| #593/#592/#591/#590 target:throne | migrate to throne_ge (THR-A-01..04), close GE-side in ENG2-P0-12 | +| #554 coordinate origin | → ENG2-P6-05 | +| #553 UiManager absolute positioning | → ENG2-P7-05 | +| #550 isometric utils | → ENG2-P8-06 | +| #548 noise | → ENG2-P8-04 | +| #547 event bus | → ENG2-P8-05 | +| #546 A* | → ENG2-P8-01 | +| #475 ALPHA-002 master / #114 ALPHA-001 | close, superseded by the new v2 master tracking issue (ENG2-P0-12) | +| #425 wasm-pack deprecation | → ENG2-P1-05 | +| #424 flipX/flipY | → ENG2-P6-02 | +| #423 drawText web | close as invalid after verify (exists at `wasm/rendering.rs:239`) — ENG2-P0-12 | +| #348/#345/#342/#340/#336/#333/#139 example games | → ENG2-P9-07 | +| #273 leak detection | → ENG2-P3-08 | +| #270 unsafe audit / #267 FFI fuzz | → ENG2-P3-09 | +| #256 perf regression detection | → ENG2-P0-05 | +| #137 F22 testing | → ENG2-P1-06 | +| #136 F21 debugger | close (shipped); authoring successor ENG2-P9-01 | +| #135 console strategy | → ENG2-P9-04 | +| #134 mobile | → ENG2-P9-03 | +| Milestones 1–9 | close as superseded after migration (ENG2-P0-12) | + +## 13. Issue count summary + +| Phase | Engine issues | Throne issues | +|---|---|---| +| P0 Instrumentation & Truth | 12 | 1 (Sync 0) | +| P1 SDK Source of Truth | 10 | — | +| P2 Render Core v2 | 12 | — | +| P3 Data Core v2 | 9 | — | +| P4 FFI v2 Convergence | 8 | 4 (Sync A) | +| P5 Parallelism & Determinism | 6 | 1 (Sync B) | +| P6 2D Render v2 | 8 | — | +| P7 Runtime Services | 6 | — | +| P8 Capability Gaps | 11 | 1 (Sync C) | +| P9 Authoring/Platforms/Polish | 9 | — | +| **Total** | **91** | **7** → **98 overall** | + +Effort mix: 9 S / 47 M / 35 L / 0 XL engine-side — every issue is sized for a single `/gh-issue --worktree` agent run producing one focused PR. Each phase doc in `docs/runbook/v2/phases/` will list the run commands per batch (issue numbers filled at creation), e.g. `Batch 2.3 Group A: /gh-issue --worktree`. + +### Critical Files for Implementation +- /Users/aramhammoudeh/dev/game/GoudEngine/goud_engine/src/libs/graphics/renderer3d/core/mod.rs (scene storage — heart of Phase 2) +- /Users/aramhammoudeh/dev/game/GoudEngine/goud_engine/src/libs/graphics/backend/wgpu_backend/frame.rs (phase timing + frame path — Phase 0/2) +- /Users/aramhammoudeh/dev/game/GoudEngine/goud_engine/src/component_ops/storage.rs (global component store to be deleted — Phase 3) +- /Users/aramhammoudeh/dev/game/GoudEngine/codegen/validate_coverage.py (source-of-truth/parity gates — Phase 1) +- /Users/aramhammoudeh/dev/game/throne_ge/docs/runbook/phase-index.md (runbook convention to mirror) diff --git a/docs/src/runbook/testing-strategy.md b/docs/src/runbook/testing-strategy.md new file mode 100644 index 00000000..8d286b50 --- /dev/null +++ b/docs/src/runbook/testing-strategy.md @@ -0,0 +1,41 @@ +# Testing & Validation Strategy (ENG2) + +You cannot safely rewrite the render core (Phase 2) or unify the component store (Phase 3) without a regression net that catches what phase counters and ad-hoc benches miss. This strategy is built in Phase 0, **before** the rewrites, and every later issue plugs into it. The foundations already exist in the repo — 5,109 tests, a null-backend, the MCP capture tools, a lavapipe lane — they are just not wired into anything that gates a merge. + +## The tiers + +1. **Unit** — pure logic, colocated `#[cfg(test)]`. The v2 render/ECS cores are designed so culling, instancing-grouping, sort-key, and propagation logic are testable **without a GPU** (math-only), per `.agents/rules/testing.md`. Testability is a design constraint, not an afterthought. + +2. **Isolation** — one subsystem behind the provider/null boundary. Renderer *logic* over `NullBackend`; ECS over a stub store; asset server over an in-memory VFS; networking/audio over null providers. Each layer gets an "isolation contract": what it must do correctly with its dependency mocked. New null/mock providers are added where a seam is missing. + +3. **Integration** — cross-layer, in-process. FFI → store → renderer command-stream assertions; the count-pinning pattern in `tests/renderer3d_frame_counts.rs` generalized (assert exact draw calls / culled / visible for a scene). + +4. **E2E** — SDK-driven headless engine (ENG2-P0-17). Each first-class SDK (C#, TypeScript, Python) boots a real headless engine, runs a scripted mini-game N frames, and asserts draw counts / world-state hash / no-crash / flat RSS. Runs in the Docker CI image on lavapipe. This is the tier that today does not exist — secondary SDK "tests" never touch a running engine. + +5. **Spec / acceptance** (ENG2-P0-16) — each behavioral issue gets a spec test at `goud_engine/tests/spec/eng2_p__.rs` encoding its Acceptance Criteria as executable assertions. The spec **is** the gate evidence. The `tdd-workflow` skill drives RED-GREEN-REFACTOR against it; the issue template's Verification section names the spec path; a reviewer confirms it was red-before-green. + +## Story gallery — "Storybook for the engine" (ENG2-P0-14) + +A declarative catalog of isolated visual scenarios with visual + metric regression, `goud_gallery`: + +- A **story** = scene setup + camera + knobs (args) + expected invariants (draw calls, instance counts, phase-timing budgets) + golden-image tag. One story per feature: each widget kind, sprite batching mode, instancing path, shadow config, post-process pass, animation state, UI layout, particle config. +- A **headless runner** renders each story to a texture on the wgpu backend (no window), captures framebuffer + metrics (MCP `capture_frame`/`get_metrics_trace`), perceptually diffs against a committed golden, and checks metric budgets. +- CI emits a **static HTML gallery** artifact per PR (image, diff-vs-golden, metrics table, pass/fail) — the Storybook browsing experience. +- **Interaction stories** (the "play function" analog) drive scripted input via the MCP `inject_input`/`replay` verbs and assert end-state. +- **Authoring**: every render/UI/particle/animation issue must add or update stories as an acceptance item; the golden-update protocol (who approves a changed golden, how baselines are versioned) is documented with the runner. + +## Determinism, property & fuzz (ENG2-P0-18) + +- **Determinism** fixture: run the headless sim 3 seeds × 10k ticks, compare world hashes. Consumed by ENG2-P5-04 and Throne's determinism gate. +- **Property** tests (`proptest`): culling visible-set vs brute force; the command-buffer decoder (Phase 4); instancing grouping. +- **Fuzz** (`cargo-fuzz`): component store add/get/remove/batch and the command-buffer decoder. Nightly lane with a time budget. + +## Infrastructure + +- **Docker CI image** (ENG2-P0-13): pinned toolchain + lavapipe + SDK toolchains; `docker compose run verify` reproduces the CI core gate locally. Metal-only real-GPU perf capture stays on the M-series host. +- **CI restructure** (ENG2-P0-19): fast core gate (fmt/clippy/unit/isolation, <10 min) always-on; heavier tiers (integration, E2E, story gallery, GPU lane, SDK matrix) staged by path filters + merge queue. +- **Self-hosted M-series nightly perf runner** (ENG2-P9-10): scenes S1–S5 + benches on real Metal; trend dashboard + regression alerting. + +## What gates a merge + +The core gate (fmt, clippy -D warnings, unit + isolation, lint-layers) is always required. `bench-gate.py`, the blocking GPU lane, per-SDK parity, and the story gallery become required as their Phase-0/Phase-1 issues land. No perf issue closes outside the [perf definition of done](perf-dod.md). diff --git a/docs/src/runbook/throne-sync.md b/docs/src/runbook/throne-sync.md new file mode 100644 index 00000000..23956d76 --- /dev/null +++ b/docs/src/runbook/throne-sync.md @@ -0,0 +1,22 @@ +# Throne Adoption & Sync Points + +Throne (the C# colony-sim at `aram-devdocs/throne_ge`) is the flagship consumer and the forcing function for the v2 perf targets. Because the engine breaks its API where necessary (no legacy code kept), Throne adopts each breaking change at defined sync points. Adoption work lives in `throne_ge` under the `goudengine-v2-adoption` milestone (label `goudengine-adoption`), not in this repo. + +Every GoudEngine `breaking-change` issue names its Throne follow-up before it closes. This is the reverse of `throne_ge/docs/runbook/engine-dependencies.md`. + +## Sync points + +| Sync | After | Throne issues | What Throne does | +|---|---|---|---| +| **Sync 0** | Phase 0 | THR-S0-01 | Bump GoudEngine; delete the `CopyGoudEngineNativeLib` workaround; re-profile GameWorld with the fixed phase counters (this becomes Phase 2's "before" evidence). | +| **Sync A** | Phase 4 gate | THR-A-01 … THR-A-04 | Bump to the v2 FFI surface; adopt batch spawn + bulk transform upload + command-buffer submission (delete per-entity `SetModelPosition`/`SetModelRotation` loops); replace the 80×80 terrain window with the engine chunked tilemap-grid primitive (full 200×200 map); re-enable shadows. | +| **Sync B** | Phase 5 gate | THR-B-01 | Replace the `Parallel.For`/job stubs with the engine job system; wire `goud_world_state_hash` into `DeterminismTests`; re-run Throne Phase 1 scale gates against engine primitives. | +| **Sync C** | Phase 8 gate | THR-C-01 | Evaluate engine A*/flow-field, event bus, and RNG/noise against Throne's C# implementations; adopt where they beat the stubs. | + +## Sync A acceptance (the prime-directive validation) + +Posted on both repos' tracking issues: Throne GameWorld, full 200×200 map, **no terrain window, shadows ON** → **≥ 120 FPS on Apple M-series**, BeginFrame+EndFrame ≤ 4 ms, and a grep gate showing zero per-entity transform-push loops. + +## Provenance + +THR-A-01 supersedes GoudEngine #596 and carries the content of the migrated GoudEngine issues #590–#593 (V2-P07…P11). Those were closed on the engine side with migration links. The engine-side prerequisites for Sync A are ENG2 Phase 2 (render core) + Phase 4 (FFI v2).