chess-gaze

Turns a chess stream video into gaze estimates and a local 3D viewer:

Video-to-gaze pipeline

When you run chess-gaze analyze <video>, it:

1. Decodes the video.
2. Finds the face, eyes, and head pose with MediaPipe.
3. Builds UniGaze inputs with the default official_geometric_v1 profile: six-point face pose, a 224 x 224 perspective-normalization warp, ImageNet normalization, and persisted geometry/asset metadata.
4. Runs UniGaze from local weights.
5. Applies each crop's inverse normalization rotation, converts the UniGaze model vector to the repository camera-ray convention, and writes calibration metadata and per-frame gaze evidence.
6. Projects gaze rays onto the 3D gaze sphere and, when configured, a calibrated screen or board target plane.
7. Builds a local browser viewer.

See the demo.

Run the analyzer

uv sync

Download the model files and put them here:

Local path	Source
models/mediapipe/face_landmarker.task	MediaPipe Face Landmarker model bundle
models/unigaze/unigaze_h14_joint.safetensors	Pinned UniGaze unigaze_h14_joint.safetensors revision
models/unigaze/face_model.txt	Pinned UniGaze six-point face-geometry source

The model registry validates exact checksums. The face-geometry asset is needed only by official_geometric_v1; the rollback profiles do not require it.

Analyze a video:

uv run chess-gaze analyze artifacts/input/test_1.mp4

Keep processed overlays for review:

uv run chess-gaze analyze artifacts/input/test_1.mp4 --save-frames

Open the printed viewer/index.html, or serve the viewer on localhost:

uv run chess-gaze view artifacts/output/<video-stem>/runs/<run-id>

Default inference expects Apple Silicon MPS, UniGaze batch size 7, and official_geometric_v1. For a portable CPU run:

uv run chess-gaze analyze video.mp4 --unigaze-device cpu --unigaze-batch-size 1

To reproduce the former direct-resize path during rollback or comparison:

uv run chess-gaze analyze video.mp4 \
  --unigaze-preprocessing-profile reference_face2x_imagenet

The geometric default restores the pinned model contract; the available streamer clips do not provide gaze targets, so this is not a measured accuracy claim. See ADR-0007.

Analysis artifacts

Runs land here:

artifacts/output/<video-stem>/runs/<run-id>/

Core outputs:

• calibration.json: analysis settings, UniGaze preprocessing metadata, and optional target-plane geometry.
• records/frames.jsonl: face, eye, head pose, UniGaze yaw/pitch, and frame errors.
• records/scene_frames.jsonl: eye points, gaze ray, sphere hit, and optional calibrated target-plane hit.
• viewer/index.html: direct-open 3D viewer with embedded scene data.
• viewer/scene-data.json: viewer data for the local server path.
• Processed frame JPEGs (if --save-frames added).

Inspect the gaze scene

The viewer shows a head model, eyes, the UniGaze ray, a gaze sphere, and translucent hit-area patches. If the run has target-plane calibration, the viewer also shows the configured screen or board plane and the current plane-hit marker. Scrub frames, play the run, switch instant or accumulated hits, and adjust display assumptions such as sphere radius and angular error.

Run artifacts stay local. The viewer loads pinned Three.js 0.185.0 modules from jsDelivr when it renders.

Runtime stack

Python 3.12, uv, PyAV, MediaPipe Face Landmarker, UniGaze, PyTorch, OpenCV, Pydantic, Pillow, and Three.js.

Development checks

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run mypy