HOI4-AI v4

A Windows-native, closed-loop AI agent that plays Hearts of Iron IV by sight.

Core principle: use the model for judgment, never for plumbing. A local vision model decides what to do (which state to build in, which tech to research) from screenshots; deterministic code handles how (hotkeys), verifies every action actually happened, and manages all plumbing (menu state, the in-game date, pause).

This is a research instrument: every cycle is logged as a replayable JSONL trace, and perception is measured offline before the live loop is trusted.

Why a rebuild

v3 was a pure open-loop, coordinate-clicking, Linux/xdotool agent that could not tell success from a softlock, swallowed every error, and was configured with a model that does not exist. v4 inverts the fragile parts. See docs/v4-design.md.

Requirements

• Windows 11, Python 3.11+ (tested on 3.14).
• Deps: pip install -e . (mss, Pillow, numpy, requests). pip install -e ".[dev]" for tests.
• A VLM runtime: Ollama (native vision) or an OpenAI-compatible server (LM Studio / llama.cpp) for grounding models. The shipped default profile is gemma4-cloud (gemma4:31b-cloud, Ollama's free cloud tier) — an interim choice to be retired for the best local profile that clears the M0 gate. Local candidates ship alongside it; see [llm.profiles.*] in config/agent.toml and run eval --all-profiles to compare.
• Hearts of Iron IV running borderless at the resolution [display] names — 3840×2160 as shipped. Calibration is resolution-specific, so the two must agree; preflight refuses to start if they don't.

Layout

hoi4_agent/      the package (errors, enums, schemas, config, geometry, preflight,
                 io, perception, brain, tools, controller, playbook, trace, eval, cli)
config/          agent.toml, playbooks/, (calibration.toml is generated)
templates/       perception template PNGs (generated by `calibrate`)
tests/           offline test suite (mocked backends + fake LLM)

Usage

# Offline (no game, no model needed):
python -m pytest                       # full test suite
python -m hoi4_agent.cli.main smoke-test --offline   # end-to-end with fakes

# Live (after installing HOI4 + Ollama):
python -m hoi4_agent.cli.main smoke-test   # validate the Windows I/O layer
python -m hoi4_agent.cli.main calibrate    # wizard: record ROIs + click-points (B=back, K=keep, S=skip)
python -m hoi4_agent.cli.main calibrate --only glyphs   # redo a subset (rois|points|tabs|templates|glyphs)
python -m hoi4_agent.cli.main eval         # M0: measure model perception on crops
python -m hoi4_agent.cli.main run          # play Germany-1936 construction + research

Glyph capture is meant to grow across sessions: the date strip renders as 12:00, 1 Jan, 1936, and the reader is all-or-nothing, so re-run calibrate --only glyphs at later in-game dates until every digit and every month name it can show has been captured. Until then the date falls back to OCR, then the VLM.

Before committing, all three checks must pass:

python -m pytest tests/ && python -m ruff check . && python -m mypy hoi4_agent

License

See LICENSE.