From 0b27136bd224f09c0ff60ab380c776e5245ec863 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Sun, 7 Jun 2026 16:01:46 +0000 Subject: [PATCH] docs: align open-world, residency, and graph compute with source - ARCHITECTURE.md: add src/world/, vk_graph_bfs, tr_bsp_stream, frame loop - DEVELOPMENT_SETUP.md + CONTRIBUTING.md: ctest matrix for open-world tests - GRAPH_COMPUTE.md: Vulkan compute enable path and troubleshooting - OPEN_WORLD.md: document nav_bake console commands - ROADMAP.md: open-world subsystem status section - AGENTS.md + CLAUDE.md: cross-reference new operational docs Co-authored-by: Tim Fox --- AGENTS.md | 4 +++- CLAUDE.md | 3 +++ docs/ARCHITECTURE.md | 37 ++++++++++++++++++++++++++++++++++++- docs/CONTRIBUTING.md | 1 + docs/DEVELOPMENT_SETUP.md | 28 ++++++++++++++++++++++++++++ docs/GRAPH_COMPUTE.md | 15 +++++++++++++++ docs/OPEN_WORLD.md | 2 ++ docs/ROADMAP.md | 8 ++++++++ 8 files changed, 96 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4468860c4..a146a2409 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,9 @@ Build artifacts go to `build-vk-Release/` and are copied to `release/`. - **Billboard / flipbook / imposter (engine-native)**: Map entities **`misc_*`** parsed on **`LoadWorld`** (`r_spriteProps 1`, auto-disabled when **`CS_ENGINE_SPRITE_META`** set). Server: **`sv_engineSprites`**, **`sv_engineSpritesSpawn`**, console **`sv_sprite_spawn`**. Client: **`cl_engineSprites`**, **`sprite_spawn`**, cgame trap **`trap_EngineSpriteAddLocal`**. QVM game traps: **`G_ENGINE_SPRITE_SHADER_INDEX`**, **`G_ENGINE_SPRITE_SPAWN`**. Lua: **`Engine.Sprites.spawnLocal` / `spawnServer`**. Demo: **`demo_sprites.cfg`**, **`lua_run demo_run_sprites()`**. - **FreeUSD (default ON)**: Git submodule **`src/external/FreeUSD`** ([gopexllc/FreeUSD](https://github.com/gopexllc/FreeUSD)); **`USE_FREEUSD=ON`** in CMake and **`./scripts/compile_engine.sh`** (auto `git submodule update --init`; pass **`nofreeusd`** to disable). Targets link **`freeusd::runtime`** + **`freeusd::c`** via **`idtech3_freeusd`**. Renderer **`r_freeusd` 1**; client **`usd_*`** tools. Init: **`./scripts/init_optional_submodules.sh --freeusd`**. See **`docs/FREEUSD.md`**. - **World districts + proxy meshes**: **`r_district` 1** (default) — USD manifest via **`district_load world/playfield.usda`**, FreeUSD snapshot parse, proxy residency (`r_districtProxy`), optional **`cm_districtStream`** sector prefetch. Console: **`district_list`**, **`district_proxy`**, **`district_load_full`**. Demo: **`exec demo_districts.cfg`**. See **`docs/DISTRICTS.md`**. -- **Infinite open worlds**: **`r_openWorld` 1** — view-driven sector residency: **`cm_stream`** + **`cm_streamMerge` 1** (overlay sector BSP collision), per-chunk **`nav/sector_X_Y.nav`**, **`sprites/sector_X_Y.ents`** billboard scatter. Console: **`openworld_start`**, **`openworld_sector`**, **`openworld_status`**. Demo: **`exec demo_openworld.cfg`**. See **`docs/OPEN_WORLD.md`**. +- **Infinite open worlds**: **`r_openWorld` 1** — view-driven sector residency: **`cm_stream`** + **`cm_streamMerge` 1** (overlay sector BSP collision), per-chunk **`nav/sector_X_Y.nav`**, **`sprites/sector_X_Y.ents`** billboard scatter. Console: **`openworld_start`**, **`openworld_sector`**, **`openworld_status`**, **`nav_bake_sector`**. Demo: **`exec demo_openworld.cfg`**. See **`docs/OPEN_WORLD.md`**. +- **World residency planner**: **`r_openWorldResidency` 1** — value-aware cardinality selection under per-layer budgets (`r_openWorldMaxSectors`, `r_openWorldMaxNavSectors`, `r_openWorldMaxSpriteSectors`) with hysteresis (`r_openWorldLoadRadius` / `r_openWorldUnloadRadius`). MP: **`sv_openWorldResidency` 1`** plans server collision; clients clamp to **`CS_ENGINE_OPENWORLD_SECTORS`**. See **`docs/WORLD_RESIDENCY.md`**. **`ctest -R test_openworld_residency`**. +- **Sector graph / GPU BFS**: **`r_graphStreamReach` 1** filters residency to k-hop reachable cells (`r_graphStreamHops`); optional **`r_graphCompute` 1`** + **`vid_restart`** for Vulkan compute BFS (`graph_bfs_status`, `graph_bfs_bench`). See **`docs/GRAPH_COMPUTE.md`**. **`ctest -R test_graph_compute`** / **`unit_sector_graph`**. - **Procedural patterns**: **`r_proc` 1** — deterministic Voronoi/grid/hex/radial/stripe/noise sector typing. Console: **`proc_pattern`**, **`proc_map`**, **`proc_sample`**. Scatter fallback: **`sprites/region_.ents`** when **`r_procScatterRegion` 1**. Demo: **`exec demo_proc.cfg`**. See **`docs/PROC_PATTERNS.md`**. - **Sector BSP fixtures**: **`python3 scripts/tools/gen_sector_bsp.py maps/sector_0_0.bsp`** — minimal collision overlay for **`cm_streamMerge`**. **`ctest -R test_cm_stream_merge`**. - **Nav sector bake**: **`nav_bake_sector 0 0`** / **`nav_bake_view`** — Recast tile from sector BSP → **`nav/sector_X_Y.nav`**. Server collision residency: **`sv_openWorld 1`**. **`ctest -R test_nav_bake`**. diff --git a/CLAUDE.md b/CLAUDE.md index 5a7827c9c..f5ee82824 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -457,6 +457,9 @@ All significant changes require architectural review: - **[QUICKSTART.md](docs/QUICKSTART.md)**: End-user quick start (download, game data, run) - **[MINIMAL_GAME_SHELL.md](docs/MINIMAL_GAME_SHELL.md)**: Smallest valid `base/` + `.pk3` bootstrap (engine-only / tech demo) - **[RELEASE_CHECKLIST.md](docs/RELEASE_CHECKLIST.md)**: Pre-release validation and release steps +- **[OPEN_WORLD.md](docs/OPEN_WORLD.md)**: Infinite open-world sector streaming (collision, nav, scatter) +- **[WORLD_RESIDENCY.md](docs/WORLD_RESIDENCY.md)**: Consistent submodular sector residency planner +- **[GRAPH_COMPUTE.md](docs/GRAPH_COMPUTE.md)**: Sector graph k-hop reachability (CPU + optional Vulkan BFS) ### Build Scripts - `./scripts/compile_engine.sh` - Primary build script diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 65943f93b..db8e10184 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -34,6 +34,12 @@ src/ ├── navigation/ Pathfinding │ ├── nav_recast.cpp/h Recast/Detour navmesh + crowd │ └── nav_bsp_extract.c BSP triangle extraction +├── world/ Open-world streaming (sector residency) +│ ├── world_open.c/h Sector load/unload coordinator (collision, nav, sprites) +│ ├── world_residency.c/h Value-aware cardinality selection under per-layer budgets +│ ├── world_district.c/h USD district → sector streaming bridge +│ ├── world_proc.c/h Procedural sector typing (region/palette) +│ └── sector_graph.c/h Grid CSR + k-hop reachability (CPU BFS, optional GPU) ├── audio/ Audio │ ├── backends/ OpenAL, SDL, null │ ├── codecs/ WAV, MP3, Opus, FLAC, WebM @@ -63,6 +69,8 @@ src/ │ │ ├── vk_volumetric_pipelines.c Volumetric fog / fluid / luminance / CBT / veg-wind pipeline setup (split from vk.c) │ │ ├── vk_volumetric_internal.c/h MSAA depth resolve, fluid sim dispatch, volumetric perf queries (split from vk.c) │ │ ├── vk_volumetric_pass_compute.c Local volumetric shadows, froxel compute, composite, SMAA (split from vk.c) +│ │ ├── vk_graph_bfs.c/h Optional Vulkan compute BFS for sector graph reachability +│ │ ├── tr_bsp_stream.c/h Streamed sector BSP visual overlay (`r_bspStream`) │ │ ├── vk_shutdown.c vk_shutdown, wait-idle, release_resources (split from vk.c) │ │ ├── vk_postfx_passes.c Bloom, SSAO/HBAO, OIT, SSR passes (split from vk.c) │ │ ├── vk_clear_attachments.c In-pass color/depth clear + dynamic color write mask (split from vk.c) @@ -116,10 +124,37 @@ CL_Frame(msec) │ ├── Choreo_Update │ ├── Horde_Update │ ├── BgMap_Frame - │ └── WinTitle_Update + │ ├── WinTitle_Update + │ └── CL_OpenWorld_Frame (when `r_openWorld` 1) + │ ├── WorldOpen_UpdateView → WorldResidency_UpdateView (when `r_openWorldResidency` 1) + │ │ └── SectorGraph_UpdateReachability (when `r_graphStreamReach` / `r_graphCompute`) + │ ├── BSP stream overlay sync (`r_bspStream`) + │ └── Billboard scatter draw for loaded sprite sectors └── Con_RunConsole() ``` +Dedicated server: `SV_OpenWorld_Frame` (when `sv_openWorld` 1) plans collision residency from player origins and publishes `CS_ENGINE_OPENWORLD_SECTORS` when `sv_openWorldSync` 1. + +## Open-world streaming + +View-driven **sector residency** for infinite worlds: modular BSP collision merge, per-chunk Detour nav tiles, and billboard scatter. Three cooperating layers: + +| Module | Role | +|--------|------| +| `world_open.c` | Sector table, layer load/unload callbacks, legacy radius disk or delegate to residency | +| `world_residency.c` | Consistent submodular cardinality selection (`r_openWorldResidency` 1) | +| `sector_graph.c` | k-hop reachability pre-filter (`r_graphStreamReach` 1); optional GPU via `vk_graph_bfs.c` | +| `cl_openworld.cpp` | Client bridge: nav tile load, scatter parse, MP configstring sync | +| `sv_openworld.c` | Server collision residency + sector list replication | +| `cm_stream.c` / `cm_stream_merge.c` | Sector pk3 prefetch and BSP brush overlay merge | + +**Planner selection** (`WorldOpen_UpdateView` in `world_open.c`): + +1. `r_openWorldResidency 0` — load all cells inside `r_openWorldRadius` (legacy disk). +2. `r_openWorldResidency 1` — `WorldResidency_UpdateView` scores candidates in a hysteresis annulus (`r_openWorldLoadRadius` / `r_openWorldUnloadRadius`), applies per-layer budgets, and bounds swaps per frame. + +Full usage, cvars, authoring, and MP rules: [OPEN_WORLD.md](OPEN_WORLD.md), [WORLD_RESIDENCY.md](WORLD_RESIDENCY.md), [GRAPH_COMPUTE.md](GRAPH_COMPUTE.md). + ## Renderer Pipeline (Vulkan) The shipping Vulkan renderer is **forward-only** with a layered HDR/post-processing pipeline. diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 5e799e1cf..2f2b0b207 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -42,6 +42,7 @@ Tests include: - `smoke_test` - Binary checks, server startup, shader validation - `check_artifacts` - Artifact format verification - `unit_macros` - PAD, PADLEN, MAX_QPATH, MAX_STRING_CHARS +- Open-world wiring: `test_openworld`, `test_openworld_sync`, `test_openworld_residency`, `test_graph_compute`, `unit_world_residency`, `unit_sector_graph` (see [DEVELOPMENT_SETUP.md](DEVELOPMENT_SETUP.md#open-world--residency--graph-compute)) ### Before Submitting diff --git a/docs/DEVELOPMENT_SETUP.md b/docs/DEVELOPMENT_SETUP.md index 334ae7c6c..c5245d894 100644 --- a/docs/DEVELOPMENT_SETUP.md +++ b/docs/DEVELOPMENT_SETUP.md @@ -294,6 +294,34 @@ cd build-vk-Release && ctest -C Release --output-on-failure ctest --preset test-vulkan-release ``` +### Open-world / residency / graph compute + +After a Release build, these script tests validate wiring without a GPU client: + +```bash +cd build-vk-Release +ctest -R 'test_openworld|test_openworld_sync|test_openworld_residency|test_graph_compute|test_cm_stream_merge|test_nav_bake' -V +``` + +| Test | What it checks | +|------|----------------| +| `test_openworld` | Sector fixtures, scatter `.ents`, `world_open.c` symbols | +| `test_openworld_sync` | MP configstring path (`sv_openworld.c`, `tr_bsp_stream.c`) | +| `test_openworld_residency` | Residency planner cvars and source wiring | +| `test_graph_compute` | `sector_graph.c` + `vk_graph_bfs.c` integration | +| `test_cm_stream_merge` | Sector BSP merge API + `gen_sector_bsp.py` layout | +| `test_nav_bake` | Recast sector tile bake from sector BSP | +| `unit_world_residency` | Cardinality selection + symmetric difference (no engine link) | +| `unit_sector_graph` | CPU BFS reachability (no engine link) | + +**Runtime collision smoke** (requires built `idtech3_server`): + +```bash +ctest -R test_openworld_runtime -V +``` + +See [OPEN_WORLD.md](OPEN_WORLD.md), [WORLD_RESIDENCY.md](WORLD_RESIDENCY.md), and [GRAPH_COMPUTE.md](GRAPH_COMPUTE.md). + ## IDE Setup ### VS Code / Cursor diff --git a/docs/GRAPH_COMPUTE.md b/docs/GRAPH_COMPUTE.md index 591cddb96..4084b822f 100644 --- a/docs/GRAPH_COMPUTE.md +++ b/docs/GRAPH_COMPUTE.md @@ -54,6 +54,21 @@ Player origin(s) → SectorGraph_UpdateReachability (CPU BFS) → WorldOpen load/unload ``` +## Vulkan compute path + +Shader: `src/renderers/vulkan/shaders/glsl/graph/graph_bfs_expand.comp` (compiled via `scripts/compile_shaders.sh` as `graph_bfs_expand_cs`). + +| Step | Action | +|------|--------| +| Enable | `set r_graphCompute 1` then **`vid_restart`** (pipeline created at renderer init) | +| Verify | `graph_bfs_status` — reports `compute=on` when `R_GraphBfs_Active` | +| Bench | `graph_bfs_bench` — 5×5 toy grid CPU vs GPU timing | +| Debug mismatch | `set r_graphStreamVerify 1` — logs CPU vs GPU bitset differences (disables per-frame cache) | + +If `graph_bfs_bench` prints `pipeline not ready`, toggle `r_graphCompute 1` and run **`vid_restart`**. GPU path is **client-only**; dedicated server uses CPU BFS. + +`r_graphBlockUnloaded 1` treats edges into sectors without loaded collision as blocked — useful when walkable corridors must follow merged BSP, not the full grid. + ## Phase 1 follow-ons (not implemented) - Nav influence heatmaps from sector reachability diff --git a/docs/OPEN_WORLD.md b/docs/OPEN_WORLD.md index d4a7a9230..edfecafc3 100644 --- a/docs/OPEN_WORLD.md +++ b/docs/OPEN_WORLD.md @@ -49,6 +49,8 @@ Demo mod: `exec demo_openworld.cfg`. | `openworld_status` | Cvars + loaded sector table | | `openworld_list` | List active sectors and layers | | `openworld_sector ` | Force-load nav + scatter (+ collision if enabled) | +| `nav_bake_sector ` | Recast-bake `nav/sector_X_Y.nav` from sector BSP collision | +| `nav_bake_view` | Bake nav tile for the sector under the current view | ## Cvars diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 4ed4f4b7e..bc88e16cd 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -65,6 +65,14 @@ Priorities that keep **CI green** and **README/build truth** aligned: - [x] Multiple model formats (glTF primary on Vulkan for GPU path; CPU tess fallback - see `docs/GLTF.md`) - [x] 6 image formats (EXR, PNG, TGA, JPG, PCX, BMP) +### Open worlds -- In progress +- [x] Sector streaming scaffold (`r_openWorld`, `cm_stream`, `cm_streamMerge`, nav tiles, scatter) +- [x] Consistent submodular residency planner (`r_openWorldResidency`, per-layer budgets) +- [x] Sector graph k-hop reachability filter + optional Vulkan compute BFS (`r_graphStreamReach`, `r_graphCompute`) +- [x] MP sector sync (`sv_openWorldSync`, `CS_ENGINE_OPENWORLD_SECTORS`) +- [ ] Streamed sector renderer VBO residency (overlay uses immediate tess path today) +- See [OPEN_WORLD.md](OPEN_WORLD.md), [WORLD_RESIDENCY.md](WORLD_RESIDENCY.md), [GRAPH_COMPUTE.md](GRAPH_COMPUTE.md) + ### Integration -- Complete - [x] All 16 systems wired into game loop - [x] 64 Lua-callable engine functions