ai_dance_game

Dance game implemented using yolo8 to detect player moves with a simple camera.

The main purpose is to remove the complex, expensive and out of production Kinect device.

Current on version 0.1. It's just a proof of concept :D

I'm happy with it because at least my girlfriend liked it. Hope someone else does too!

Godot version (in progress)

The game is being migrated from pygame to a Godot 4.x front-end. Because Godot's CameraServer has no Linux/Windows desktop backend, webcam capture and YOLOv8 pose detection stay in Python and stream keypoints to Godot over UDP:

ai_camera_server/vision_service.py  ──(UDP keypoints, :5005)──▶  Godot client (scenes/ + scripts/)
  webcam + YOLOv8                                                  menus, audio, scoring, rendering

Project layout

• ai_camera_server/ — headless Python vision service (webcam + YOLOv8 → UDP keypoints)
• game/ — shared Python pose code (keypoint extraction, geometry)
• choreography/ — dance.csv/dance.json plus tools to build them (extract_video.py, extract_poses.py)
• scenes/, scripts/ — the Godot 4.x client
• test/ — udp_client.py to inspect the keypoint stream
• songs/ — drop .mp3 songs here; they appear in the in-game recorder
• weights/ — the YOLOv8 pose model

Running it

Run the Python commands from the repository root (with your virtualenv active) so the game package and the asset paths resolve.

1. Install Python deps: pip install -r requirements.txt

2. Start the vision service (keep it running): python -m ai_camera_server.vision_service

   • Debug preview window: python -m ai_camera_server.vision_service --show
   • Sanity-check the stream without Godot: python test/udp_client.py

3. Open the project (this folder) in the Godot 4.x editor and press Play, or run godot --path . from the command line. Pick 1 Player or 2 Players (stand side by side for two), then choose a choreography to dance to. The game waits until everyone is in frame, counts you in, then plays the choreography's song. The round ends when the song stops (capped at 30s for testing) and a winner is shown.

   Or pick Record Choreography to author a dance live (see below).

Run it with one command

./run.sh starts the Python vision sidecar and the Godot game (fullscreen), and stops the sidecar when you quit. It runs the project directly with a Godot binary, so the data folders (songs/, choreography/) stay writable — recording and dropping in new .mp3s just work. Override defaults with env vars: GODOT_BIN, VENV, WEIGHTS.

GODOT_BIN=/path/to/godot ./run.sh

Deploy to a Jetson Orin Nano (TV setup)

The game is a Godot front-end + a Python ML sidecar; on the Jetson the sidecar must use the GPU. Steps:

1. Flash JetPack 6.x (gives you CUDA/cuDNN/TensorRT).
2. Python deps with Jetson-compatible wheels — the default pip torch/torchvision do not work on ARM64; install NVIDIA's prebuilt wheels, then ultralytics + opencv-python. Follow the official Ultralytics Jetson guide. Put them in a venv (e.g. ./ai_dance_game).
3. Convert the model to TensorRT once for real-time inference: yolo export model=weights/yolov8m-pose.pt format=engine half=True (consider yolov8s-pose/yolov8n-pose for more headroom).
4. Get a Godot arm64 binary for the Jetson (Godot Linux downloads).
5. Run: GODOT_BIN=/path/to/godot.arm64 WEIGHTS=weights/yolov8m-pose.engine ./run.sh

Connect a USB webcam, HDMI to the TV (display + audio), and you're ready to play. Quit from the in-game Quit button or Ctrl-C in the terminal (the sidecar is cleaned up either way).

Choreography

The choreography is a time-indexed list of pose keyframes that the game interpolates between, so the reference figure moves continuously (no photos or video ship — only keypoints). Two sources, loaded in this order:

• choreography/dance.csv (preferred) — a dense per-frame timeline recorded from a dance video. One row per frame; time = row index / fps; metadata in the top comment.
• choreography/dance.json (fallback) — the older sparse keyposes, interpolated.

A timeline carries its own song in the CSV metadata (# song=...), so gameplay plays the right track; otherwise it falls back to the default song. There are two ways to make one:

Record it live in-game (easiest). Drop .mp3 files into songs/, then on the menu choose Record Choreography. Your right hand is the cursor — rest it on a button (on the right edge, far from the centre) for a moment to activate. Pick a song, press Play, get counted in, and dance; your pose is saved to choreography/<song-name>_N.csv (auto-numbered per song, e.g. My_Song_1.csv, My_Song_2.csv) synced to that song. All recorded .csvs show up on the song-select screen when you start a game.

Extract from a video (offline).

1. Put the video in a local pose_sources/ folder (gitignored).
2. Run python -m choreography.extract_video --video pose_sources/dance.mp4 --fps 15 \ --song "res://songs/your-song.mp3".
3. Commit the generated choreography/dance.csv. The video stays local.

Players are scored on limb orientation against the best-matching reference pose within a small time window (±0.3 s), so reaction lag doesn't unfairly tank the score. The live score is a 0–100 match; in 2-player mode the higher average wins.

(extract_poses.py still exists to build the sparse dance.json from still images.)

Demo video

output2.mp4

Test Song credits:

Song: Cartoon, Jéja - On & On (feat. Daniel Levi) [NCS Release] Music provided by NoCopyrightSounds Free Download/Stream: http://ncs.io/onandon Watch: http://youtu.be/K4DyBUG242c

Big thanks for the really great song!

Future improvements:

• Make score points more fair
• Improve comparison between player pose and choreography move. It only measure distance in the current state.
• Add multi player mode
• Improve game play interface. It's too simple and not very exciting
• Improve game menus
• Add start screen
• Improve camera resolution handling
• Test with full body detection. Maybe buy a better camera
• Pack it so other people can use it.