Dev

.github/workflows/ci.yml

@@ -0,0 +1,86 @@
+name: CI
+
+on:
+  push:
+    branches: [main, master, dev]
+  pull_request:
+    branches: [main, master, dev]
+
+concurrency:
+  group: ci-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lint:
+    name: Ruff
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+          cache-dependency-path: requirements.txt
+
+      - name: Install linters
+        run: pip install "ruff>=0.1"
+
+      - name: Ruff check
+        run: ruff check src tests scripts
+
+  build:
+    name: Package build
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+          cache-dependency-path: requirements.txt
+
+      - name: Install build backend
+        run: pip install build
+
+      - name: Build sdist and wheel
+        run: python -m build
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/*
+
+  test:
+    name: Tests (Python ${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+          cache-dependency-path: requirements.txt
+
+      - name: Install system dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y libgl1 libglib2.0-0 libsdl2-2.0-0
+
+      - name: Install dependencies
+        run: |
+          python -m pip install -U pip
+          pip install -r requirements.txt
+          pip install -e .
+
+      - name: Pytest
+        env:
+          # Headless / no display for pygame in CI
+          SDL_VIDEODRIVER: dummy
+        run: pytest -q --tb=short

.github/workflows/release.yml

@@ -0,0 +1,34 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*"
+
+permissions:
+  contents: write
+
+jobs:
+  build-and-attach:
+    name: Build and attach wheels
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+          cache-dependency-path: requirements.txt
+
+      - name: Install build
+        run: pip install build
+
+      - name: Build sdist and wheel
+        run: python -m build
+
+      - name: Publish GitHub release assets
+        uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*
+          generate_release_notes: true

.gitignore

@@ -8,3 +8,7 @@ checkpoints/
 wandb/
 outputs/
 .hydra/
+*.egg-info
+dist/
+build/
+.cursor/

README.md

@@ -1,16 +1,12 @@
-# Village AI War — 2D Hierarchical RL Environment
+# Village AI War
 
-Custom Gymnasium environment for hierarchical multi-agent reinforcement learning. The **baseline is fully RL-driven**: low-level units are controlled by a learned policy (no movement heuristics), and both teams can be trained with **self-play** against pools of past checkpoints.
+A **Gymnasium** environment for a small RTS-style village game. You can watch a quick demo, play against a trained **MAPPO** bot team, or train your own policies.
 
-## Architecture
+Training in this repo is **MAPPO only** (multi-agent PPO with a shared critic). Everything else is the game simulation, rendering, and helpers.
 
-- **Bot agents (low level):** One **role-conditioned** PPO policy (`RoleConditionedPolicy`) — shared backbone plus a learned role embedding from the observation one-hot (see `BotObsBuilder`). All roles share weights.
-- **Village agent (high level):** Strategic manager — **MaskablePPO** with invalid-action masking (`MultiInputPolicy` on dict observations).
-- **Self-play:** Stage 1 samples opponents from `checkpoints/pool/bots/`; stage 2 samples village opponents from `checkpoints/pool/village/`. Empty pools fall back to random opponent actions.
-- **Unified training:** One Hydra entry point (`training=train_unified`, `training.stage=0`) alternates PPO on bots and MaskablePPO on the red manager. Each environment step matches village self-play order (all bots act, then both managers). Blue bots and blue manager are sampled from the same pools; the non-training partner on red is frozen from the last saved checkpoint until the next phase.
-- **Reward shaping:** Dynamic global reward modes controlled by the village agent (unchanged).
+---
 
-## Quick Start
+## Install and first run
 
 ```bash
 cd VillageAI_War
@@ -20,142 +16,54 @@ pip install -e .
 python scripts/run_game.py
 ```
 
-### Watching trained policies
+That opens a **random demo**: the AI village manager picks valid actions at random (2D window). Add `--human-3d` for a passive 3D view (needs working OpenGL; on Linux/WSL you may need `libgl1` / Mesa — if 3D fails, the script falls back to 2D).
 
-[`scripts/run_game.py`](scripts/run_game.py) can load the same checkpoints produced by training. If a path is missing or fails to load, that component falls back to random valid actions (same as the old demo).
+---
 
-```bash
-# Defaults: checkpoints/village/village_final.zip, checkpoints/bots/bot_final.zip
-python scripts/run_game.py
-
-# Explicit paths and deterministic manager actions
-python scripts/run_game.py \
-  --village-checkpoint checkpoints/village/village_final.zip \
-  --opponent-village-checkpoint checkpoints/pool/village/village_iter5.zip \
-  --bot-checkpoint checkpoints/bots/bot_final.zip \
-  --deterministic --seed 42 --max-steps 2000
-
-# Artifacts from unified training
-python scripts/run_game.py \
-  --village-checkpoint checkpoints/unified/village_final.zip \
-  --bot-checkpoint checkpoints/unified/bot_final.zip \
-  --deterministic --seed 42
-```
-
-When any village, opponent, or bot checkpoint is loaded (or you pass a path that exists), the viewer runs **both** village managers each tick (trained or random per side), then resolves the tick in the same order as self-play training (`run_bots_then_village_decisions` in [`GameEnv`](src/village_ai_war/env/game_env.py)). If **no** checkpoints load, behavior matches the legacy script: one random manager action per step for team 0 only.
-
-The pygame **human** window includes numeric **row/column axes**, a **legend** (terrain colors, harvest hints `w`/`s`/`f`, unit roles and team rings, building abbreviations), and a **bottom HUD** (tick, winner, resources, population, global mode per team). `rgb_array` mode is still the raw map only for headless frames.
-
-**3D view** (OpenGL via [moderngl](https://github.com/moderngl/moderngl)): run `python scripts/run_game.py --human-3d` for an extruded terrain board, **building silhouettes** (town hall roof, tower spire, farm silo, etc.), **bots as team disk + role sphere + team ring**, and a **Russian legend panel** on the right (including a **live HUD**: tick, wood/stone/food, alive bots / pop cap, village AI mode per team). **Camera:** left-drag on the map to orbit (yaw/pitch); arrow keys nudge the camera; optional idle spin via `auto_rotate_deg_per_sec` (default `0`). Tuning: `window_width_3d`, `legend_width_3d`, `camera_dist_scale`, `orbit_mouse_sensitivity`, `orbit_key_deg_per_sec`, … in [`configs/default.yaml`](configs/default.yaml). In code, use `GameEnv(..., render_mode="human_3d")` or `render_mode="rgb_array_3d"` for full-window RGB captures (map + legend).
-
-On **Linux**, Mesa packages expose `libGL.so.1` (not `libGL.so`); this project patches library loading so moderngl finds them. If you still see GL errors, install dri drivers: `sudo apt install -y libgl1-mesa-dri libegl1`. On **WSL2** you need a working GUI stack (WSLg); without it, use 2D or run from native Windows. If 3D fails, `run_game.py` falls back to the 2D pygame window and logs a warning.
-
-### Training (Hydra)
-
-The default config composes `training: train_bots_selfplay` (see [`configs/default.yaml`](configs/default.yaml)). Stages are selected with `training.stage` (`0` = unified, `1`–`3` = legacy pipeline).
-
-**Stage 1 — bot self-play (role-conditioned PPO)**
-
-```bash
-python scripts/run_training.py training.stage=1
-```
+## Play against MAPPO
 
-Uses `game.initial_bots: 1` in [`configs/training/train_bots_selfplay.yaml`](configs/training/train_bots_selfplay.yaml) so each team has one controllable unit per episode (reproducible with `SubprocVecEnv`; multi-bot stage 1 would need extra machinery such as checkpoint sync or `DummyVecEnv`-only runs).
-
-**Stage 2 — village self-play (MaskablePPO, frozen bot policy)**
-
-Requires a stage-1 artifact at `checkpoints/bots/bot_final.zip`.
+You control **blue**; **red** uses your checkpoint. Human play is **2D only** (MAPPO training matches this).
 
 ```bash
-python scripts/run_training.py training=train_village_selfplay training.stage=2
+python scripts/run_game.py \
+  --mappo-opponent checkpoints/bots_mappo/mappo_bot_final.zip \
+  --seed 0 --max-steps 500
 ```
 
-**Stage 3 — joint fine-tuning**
+Use a checkpoint trained with the same **map size** and **`game.max_bots_for_role_change`** as in your config.
 
-Loads `checkpoints/village/village_final.zip` when present; bots use `checkpoints/bots/bot_final.zip` when present (otherwise random bot actions).
+---
 
-```bash
-python scripts/run_training.py training=train_joint training.stage=3
-```
-
-**Unified training (recommended)** — `training.stage=0`, single process alternating two learners
-
-No prerequisite checkpoints. Team 0’s bot policy (PPO) and red manager (MaskablePPO) are updated in turns; after each phase the trainer saves so the other phase loads a frozen partner from disk. Opponents use the same self-play pools as stages 1–2; empty pools still fall back to random valid actions.
+## Train MAPPO
 
 ```bash
-python scripts/run_training.py training=train_unified
+python scripts/run_training.py
 ```
 
-You can also set `training.stage=0` with another `training=` group; built-in defaults for `unified.*` still apply, but prefer `training=train_unified` so values in [`configs/training/train_unified.yaml`](configs/training/train_unified.yaml) are loaded.
-
-Configure macro steps with `unified.bot_steps_per_turn`, `unified.village_steps_per_turn`, `unified.n_cycles`, optional `unified.first_phase` (`bot` or `village`), and `unified.push_to_pool` (append snapshots to `checkpoints/pool/bots` and `.../village`) in [`configs/training/train_unified.yaml`](configs/training/train_unified.yaml). Each phase logs an estimated **SB3 iteration count** (`n_steps × n_envs` env-steps per iteration). `unified.progress_bar: true` enables Stable-Baselines’ tqdm progress bar for the current `learn()` (requires `tqdm` and `rich` in [`requirements.txt`](requirements.txt)). `unified.sb3_verbose` controls SB3’s stdout tables (`0` by default when using the progress bar). `unified.progress_log_interval_sec` throttles **Loguru** lines with progress, ETA, env-steps/s, and wall time of the previous full SB3 iteration (see `run_training.log` under the Hydra run directory). Set `unified.plot_metrics_on_finish: true` to write PNG grids of TensorBoard scalars into the same Hydra output folder when the run finishes (or run [`scripts/plot_tensorboard_scalars.py`](scripts/plot_tensorboard_scalars.py) manually). Outputs live under `checkpoints/unified/` (`bot_final.zip`, `village_final.zip`, plus per-cycle checkpoints and `bot_latest` / `village_latest` stems used between phases).
-
-The bot phase uses `DummyVecEnv` only so every sub-env shares the in-process `bot_policy_holder`; do not use `SubprocVecEnv` for that phase. Default `game.initial_bots: 1` matches stage 1; more red bots require a live model in the holder for the extra units.
-
-Stages 1–3 remain a supported alternative pipeline.
-
-**Useful overrides**
+Quick test (small run):
 
 ```bash
-python scripts/run_training.py training.total_timesteps=2000 training.n_envs=1
+python scripts/run_training.py \
+  training.total_timesteps=2000 training.n_envs=1 training.selfplay_iterations=2
 ```
 
-### Metrics (TensorBoard)
+Settings live in [`configs/training/train_mappo_bots.yaml`](configs/training/train_mappo_bots.yaml) (merged via [`configs/default.yaml`](configs/default.yaml)). Override anything under `training.*` on the command line.
 
-With `logging.use_tensorboard: true` (default) and `tensorboard` installed, stage 1 and 2 write training scalars under `logs/bots/` and `logs/village/`; unified training writes under `logs/unified_bots/` and `logs/unified_village/`. Periodic evaluation logs `eval/mean_reward` (and related fields) under `logs/bots_eval/` and `logs/village_eval/` when `training.eval_freq > 0` (default `10000` **environment timesteps** between evals; internally scaled by `n_envs` per Stable-Baselines3). The unified config sets `eval_freq: 0` by default (no separate eval pass yet). Tune with `training.n_eval_episodes`.
+**TensorBoard:** logs go under `logs/mappo_bots/` when enabled. Then:
 
 ```bash
 tensorboard --logdir logs/
 ```
 
-After unified training (or anytime), generate static PNG grids from the latest run in each subfolder:
+**Plots:** `python scripts/plot_tensorboard_scalars.py --log-root logs` writes `logs/plots/mappo_bots_scalars.png` by default.
+
+---
 
 ```bash
-python scripts/plot_tensorboard_scalars.py --log-root logs
+python scripts/evaluate.py
 ```
 
-Default outputs: `logs/plots/unified_bots_scalars.png` and `logs/plots/unified_village_scalars.png` (or under the Hydra run directory when `unified.plot_metrics_on_finish` is enabled).
+Runs many episodes with random village actions and prints rough stats.
 
-**Best vs last checkpoint:** when evaluation is enabled and at least one eval run produced a best model, `checkpoints/bots/bot_final.zip` and `checkpoints/village/village_final.zip` are copies of the best eval checkpoint (also saved as `bot_best.zip` / `village_best.zip`). The last weights after all self-play iterations are kept as `bot_last.zip` / `village_last.zip`. Set `training.eval_freq=0` to keep the previous behavior (final = last iteration only). Stage 3 joint training does not add a separate eval pass yet; it still saves `checkpoints/joint/joint_final.zip` from the end of the run. Unified training always ends with `bot_final.zip` / `village_final.zip` as the last full save after all cycles (no best-model selection unless you add eval later).
 
-**Evaluation**
-
-```bash
-python scripts/evaluate.py
-```
 
-## Checkpoints (typical layout)
-
-| Path | Contents |
-|------|-----------|
-| `checkpoints/bots/bot_final.zip` | Stage 1 policy (best eval mean reward when `training.eval_freq > 0`, else last iteration) |
-| `checkpoints/pool/bots/*.zip` | Historical bot policies for self-play |
-| `checkpoints/village/village_final.zip` | Stage 2 manager (same best-vs-last rule as bots) |
-| `checkpoints/pool/village/*.zip` | Historical village policies for self-play |
-| `checkpoints/joint/joint_final.zip` | Stage 3 output |
-| `checkpoints/unified/bot_final.zip` | Unified loop bot policy (last save after all cycles) |
-| `checkpoints/unified/village_final.zip` | Unified loop village policy (last save after all cycles) |
-| `checkpoints/unified/bot_latest.zip` / `village_latest.zip` | Latest weights exchanged between alternating phases |
-| `checkpoints/unified/bot_cycle*.zip` / `village_cycle*.zip` | Periodic `CheckpointCallback` snapshots during unified runs |
-
-## Project layout (RL baseline)
-
-- [`src/village_ai_war/env/game_env.py`](src/village_ai_war/env/game_env.py) — `step`, `step_with_opponent`, `step_village_only`, optional `game.bot_rl_checkpoint` / `training.bot_checkpoint` for frozen bot PPO
-- [`src/village_ai_war/models/role_conditioned_policy.py`](src/village_ai_war/models/role_conditioned_policy.py)
-- [`src/village_ai_war/training/self_play_env.py`](src/village_ai_war/training/self_play_env.py) — `SelfPlayBotEnv`, `SelfPlayVillageEnv`, `UnifiedBotSelfPlayEnv`
-- [`src/village_ai_war/training/train_bots_selfplay.py`](src/village_ai_war/training/train_bots_selfplay.py), [`train_village_selfplay.py`](src/village_ai_war/training/train_village_selfplay.py), [`train_joint.py`](src/village_ai_war/training/train_joint.py), [`train_unified.py`](src/village_ai_war/training/train_unified.py), [`tensorboard_plots.py`](src/village_ai_war/training/tensorboard_plots.py), [`plot_tensorboard_scalars.py`](scripts/plot_tensorboard_scalars.py)
-
-Legacy trainers [`train_bots.py`](src/village_ai_war/training/train_bots.py) and [`train_village.py`](src/village_ai_war/training/train_village.py) are not used by [`scripts/run_training.py`](scripts/run_training.py).
-
-## Project status
-
-- [x] Data structures (Pydantic)
-- [x] Map generator
-- [x] Economy / combat / building systems
-- [x] Observation builders and action masker
-- [x] GameEnv (Gymnasium), no bot heuristics
-- [x] Role-conditioned bot policy + PPO self-play (stage 1)
-- [x] Village MaskablePPO self-play with RL bots (stage 2)
-- [x] Joint fine-tuning with RL bots (stage 3)
-- [x] Unified training loop (alternating bot PPO + village MaskablePPO)
-- [x] Pygame renderer

configs/buildings.yaml

@@ -1,31 +1,37 @@
 # @package _global_
 buildings:
   townhall:
-    hp: 1000
+    hp: 1500
     cost: {}
   barracks:
     hp: 300
     cost:
-      wood: 100
+      wood: 80
+    construction_ticks: 24
   storage:
     hp: 200
     cost:
-      wood: 50
+      wood: 40
+    construction_ticks: 16
   farm:
     hp: 200
     cost:
-      wood: 80
+      wood: 60
+    construction_ticks: 20
   tower:
-    hp: 400
+    hp: 350
     cost:
-      stone: 100
+      stone: 80
+    construction_ticks: 28
   wall:
-    hp: 500
+    hp: 400
     cost:
-      stone: 30
+      stone: 25
+    construction_ticks: 22
   citadel:
     hp: 800
     cost:
-      stone: 200
-      wood: 150
+      stone: 150
+      wood: 100
+    construction_ticks: 40
   citadel_pop_bonus: 5

configs/combat.yaml

@@ -3,19 +3,19 @@ combat:
   stats:
     warrior:
       hp: 100
-      damage: 25
+      damage: 20
       attack_range: 1
     gatherer:
-      hp: 80
-      damage: 8
-      attack_range: 1
-    farmer:
       hp: 70
       damage: 5
       attack_range: 1
+    farmer:
+      hp: 60
+      damage: 3
+      attack_range: 1
     builder:
-      hp: 80
-      damage: 8
+      hp: 75
+      damage: 5
       attack_range: 1
-  tower_damage: 15
+  tower_damage: 12
   tower_range: 3