Head-coupled perspective for Minecraft. A webcam tracks your head; the projection is rebuilt each frame as an off-axis (asymmetric) frustum, so the monitor stops being a screen and starts being a window onto a 3D diorama.
Client-side Fabric mod for Minecraft 26.2, plus a standalone Python head tracker. Nothing is synced and no server component exists — the projection is purely a property of how this machine's screen is being looked at.
Working end to end: head movement drives the projection in game.
| Part | State |
|---|---|
| Fabric mod (26.2) | ✅ builds and runs |
| UDP transport | ✅ verified in game, including malformed input |
| Off-axis projection | ✅ verified in game |
| Eye translation (parallax) | ✅ implemented, confirmed working |
| Toggle key, cull widening, view-bob suppression | ✅ verified in game |
| In-game calibration panel | ✅ implemented |
| Python head tracker | ✅ runs; not yet calibrated |
| Diorama scene, fixed viewpoint | ❌ not started |
| Packaging for non-developers | ❌ not started |
Everything marked ✅ was observed working at runtime, not merely compiled. Calibration constants in the tracker are still hardcoded for one particular setup — see Limitations.
- JDK 25 — required by Minecraft 26.2.
- Python 3.12+ — verified working on 3.14.6. Note that mediapipe's PyPI classifiers claim 3.9–3.12 only; that metadata is stale and the resolver disagrees with it.
- A webcam.
- Gradle is not needed; the committed wrapper fetches it.
Two processes. Start them in either order — UDP simply drops packets when nothing is listening.
# Terminal 1: the tracker (leave running)
cd tracker
python -m venv .venv
.venv\Scripts\python.exe -m pip install -r requirements.txt
.venv\Scripts\python.exe head_tracker.py
# Terminal 2: the game
./gradlew runClientThe model weights are not in the repo (3.6 MB binary). Fetch once into tracker/models/:
curl -o models/face_landmarker.task https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.taskCalling the venv's interpreter directly is deliberate: .venv\Scripts\activate fails on a
default Windows PowerShell with PSSecurityException. Activation buys nothing here, so
there is no reason to relax a system-wide security policy for it.
| Key | Action |
|---|---|
| F8 | Toggle the diorama view |
| F10 | Open/close the calibration panel |
The HUD and the held item hide with the view — renderItemInHand is gated on the same
flag, so one switch covers both.
The F10 panel adjusts everything live while the world keeps rendering behind it, so you
can watch the effect while dragging. Changes apply in memory only; Save to file writes
them to config/terrarium.json.
| Setting | Meaning |
|---|---|
fovScale |
Widens the frustum beyond what the geometry justifies. See below. |
worldScale |
Blocks of camera movement per centimetre of head movement. Higher means stronger depth but multiplies tracker jitter equally. |
referenceEyeZCm |
Your habitual viewing distance. Only deviations from it move the camera along the view axis. |
screenWidthCm / screenHeightCm |
Physical visible image area, excluding bezel. Read these off the monitor's EDID rather than measuring — a hand measurement that disagrees with the pixel aspect ratio silently stretches the scene. |
cullFrustumFovDegrees |
How generously geometry survives culling. 120° confirmed sufficient. |
An honest head-coupled projection shows the field of view your screen actually subtends. A 19 cm tall panel viewed from 50 cm subtends 21.5° vertically. Minecraft's ~70° default is a wide-angle exaggeration unrelated to where you sit, so the correct projection feels drastically zoomed by comparison.
This is geometry, not a bug. The three ways out, in descending order of honesty:
- Use a bigger screen. The real fix. Field of view scales with screen size.
- Sit closer. Correct, if uncomfortable.
- Raise
fovScale. A deliberate cheat: it pretends the screen is larger than it is. The illusion depends on rendered angles matching the angles your eye subtends, so at high values the sense of looking through the screen degrades into looking at a moving picture. Where that line falls is a matter of taste.
Two measurements dominate depth accuracy, both currently constants in
tracker/head_tracker.py:
- Focal length —
head_tracker.py --calibratemeasures it. The FOV-derived fallback is only a guess. - Interpupillary distance —
INTERPUPILLARY_DISTANCE_CM. Every depth estimate scales with it, so a 5% error becomes a 5% error in every reportedz.
Also set CAMERA_OFFSET_Y_CM to how far the webcam sits above the centre of the screen —
roughly half the screen height plus bezel. Getting it wrong tilts the whole illusion.
One JSON object per datagram to 127.0.0.1:5005, centimetres throughout. The origin is
the centre of the visible screen area — not the webcam, not the bezel.
{"x": -3.4, "y": 5.1, "z": 62.0}x positive to your right, y positive up, z perpendicular distance from the screen
plane. Non-positive z is rejected as degenerate.
tracker/udp_listener.py mimics the mod's receiver, so the tracker can be verified with
no JDK and no Minecraft involved.
- The tracker is calibrated for one setup. Focal length, interpupillary distance and
camera offset are constants in a
.pyfile. Anyone else's numbers will be wrong, and a wrong depth estimate looks like a broken effect rather than a wrong setting. - No diorama scene yet. The intended end state is a small fixed diorama with the player camera pinned to a viewpoint in front of it. Currently you play normally.
- Not packaged. Running this requires a JDK, Python, a virtualenv and a manual model download. See the discussion of distribution in the project history.
Things that cost real time to discover and are easy to get wrong again.
No mappings line in build.gradle. From 26.x Minecraft ships unobfuscated and Loom
takes no mappings input. Adding officialMojangMappings() or Yarn is 1.21.x-era advice
and breaks the build.
The projection matrix is built in Camera, not GameRenderer. In the 26.x
render-state architecture Camera.extractRenderState bakes it into
CameraRenderState.projectionMatrix, and GameRenderer.renderLevel merely copies that
field. Hooking GameRenderer per older tutorials targets the wrong place.
Minecraft uses reversed-Z. Projection.getMatrix passes the far distance as JOML's
near argument and vice versa. OffAxisProjection mirrors that swap and measures the
frustum extents on the matching plane. The intuitive ordering renders the scene sorted
back-to-front, and fails silently rather than loudly.
Both the frustum shear and the eye translation are required. The shear alone only skews the image, which reads as distortion. Parallax — near things sliding further than far things — is what the brain reads as depth.
The frustum projects through the rendered area, not the panel. Those coincide only in fullscreen. The configured centimetres are converted to a per-pixel size via the monitor resolution and scaled by the framebuffer, which locks the frustum aspect to the render aspect and makes stretching impossible in any window shape.
Culling needs the shifted eye position too. Camera.update prepares the cull frustum
from the unshifted position. Widening its angle does not compensate, because the error is
a translation — at high worldScale it reaches several blocks and culls visible geometry.
API names that moved since 1.21.x: fabric-key-mapping-api-v1 / KeyMappingHelper
(not KeyBindingHelper), KeyMapping.Category is a record over Identifier (not a
String), Options.hideGui is gone (use Hud.toggle()), screens are driven through
Minecraft.gui.screen() / gui.setScreen(), and mp.solutions.face_mesh was removed
from mediapipe entirely in favour of the Tasks API.
| Component | Version |
|---|---|
| Minecraft | 26.2 |
| Fabric Loom | 1.17.16 |
| Fabric Loader | 0.19.3 |
| Fabric API | 0.155.2+26.2 |
| Gradle | 9.5.1 |
| Java | 25 |
| mediapipe | 0.10.35 |
MIT — see LICENSE.