fix(frames): apply display-matrix rotation when decoding frames

[AmitMY]

Problem

Vertical (phone-recorded) videos came out 90° rotated: frames were decoded in their stored landscape orientation and metadata reported coded dimensions. PyAV does not apply the container's display-matrix rotation side data (unlike the ffmpeg CLI, which autorotates by default). Downstream, this produced sideways pose estimations in pose-estimation.

Changes

• frames.py: new _frame_to_rgb() rotates decoded frames to display orientation using VideoFrame.rotation. Direction verified empirically: rotation=90 + np.rot90(k=1) matches ffmpeg autorotate pixel-exactly (the opposite direction differs by a mean of ~68).
• metadata.py: VideoMetadata.width/height are now display-oriented (swapped for 90°/270°), consistent with the frames the package yields. New rotation field (default 0, appended last — backward compatible with positional unpacking). Rotation is probed by decoding one frame and rewinding, since PyAV only exposes it per-frame.
• read_frames_from_stream: input may be non-seekable (e.g. an OS pipe in pose-estimation's /stream endpoint), so the first frame is decoded eagerly for the rotation and replayed through the generator; skip_frames semantics unchanged.
• pyproject.toml: pin av>=14.1 (VideoFrame.rotation was added in PyAV 14.1).

Tests

New tests/test_rotation.py (7 tests) with a 78KB asset tests/assets/rotated90.mp4 — a downscaled clip of a real vertical sign-language recording (stored 640×360, rotation=90):

• metadata reports portrait dims + rotation (path and bytes variants)
• unrotated video keeps rotation == 0
• frames decode portrait via read_frames_exact
• pixel comparison against ffmpeg autorotate output — catches wrong rotation direction, which shape checks alone cannot
• stream path: rotated frames, display-oriented metadata, and skip_frames with the eagerly-decoded first frame

ruff check clean, 45 tests pass locally. (3 pre-existing test_regression.py failures on my machine also fail on clean main — local ffmpeg 8 vs PyAV's bundled FFmpeg differ by ±3 in YUV→RGB; unrelated.)

Note for consumers

video_metadata() now reports display dimensions rather than raw ffprobe-style coded dimensions. For pose-estimation this is the desired behavior: the pose header and landmark unnormalization stay consistent with the rotated frames. Poses previously extracted from rotated videos are sideways and should be re-extracted.

🤖 Generated with Claude Code

---

Note

Medium Risk
Behavior change for rotated videos (frame pixels and metadata dimensions) and a breaking semantic shift for consumers expecting raw ffprobe-coded sizes; stream path now decodes one frame up front.

Overview
Fixes phone-recorded / display-matrix rotated videos where PyAV decoded frames in stored orientation while ffmpeg autorotates by default.

Frame decoding now routes through _frame_to_rgb(), applying VideoFrame.rotation with np.rot90 and a C-contiguous copy so OpenCV/MediaPipe accept the arrays. VideoMetadata reports display width/height (swapped for 90°/270°), adds a rotation field, and probes rotation by decoding the first frame when needed. read_frames_from_stream eagerly decodes the first frame for rotation on non-seekable inputs (pipes), replays it through the generator, and preserves skip_frames behavior.

Tooling: requires Python ≥3.10, pins av>=14.1, drops 3.8/3.9 from CI, and adds tests/test_rotation.py (ffmpeg pixel checks, pipe streaming).

^{Reviewed by Cursor Bugbot for commit 5a7a53e. Bugbot is set up for automated code reviews on this repo. Configure here.}

Summary by CodeRabbit

• New Features

  • Video frames are now automatically rotated to display orientation based on embedded rotation metadata (e.g., from mobile recordings)
  • Added rotation angle (in degrees) to video metadata reporting
  • Width and height dimensions now reflect the display orientation

• Chores

  • Updated av library dependency requirement to version 14.1+

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds complete support for display-matrix rotation in video frames: VideoMetadata now includes a rotation field and probes it from the first frame, frames are converted to display orientation via rotation, and comprehensive tests validate behavior against ffmpeg's autorotate output.

Changes

Display-Matrix Rotation Support

Layer / File(s)	Summary
Dependency and Metadata Contract pyproject.toml, simple_video_utils/metadata.py	av>=14.1 is pinned to ensure VideoFrame.rotation availability. VideoMetadata adds a rotation: int = 0 field documenting display rotation in degrees.
Rotation Detection and Normalization simple_video_utils/metadata.py	New _probe_rotation helper decodes the first frame to extract rotation, handles errors by returning 0, and rewinds. video_metadata_from_container accepts optional pre-known rotation, probes when omitted, normalizes to [0, 360), and swaps width/height for 90°/270° orientations.
Frame Conversion and Display Orientation simple_video_utils/frames.py	New helper converts PyAV frames to RGB with per-frame rotation applied. _generate_frames uses this helper to yield display-oriented arrays. read_frames_from_stream eagerly decodes the first frame to obtain rotation metadata and forwards it, then yields the pre-decoded frame (unless skipped) followed by remaining frames from the container.
Comprehensive Rotation Test Suite tests/test_rotation.py	New test module with fixtures for rotated/non-rotated test videos, ffmpeg_autorotated_frames helper to generate reference frames via CLI, and seven test methods validating metadata dimensions/rotation reporting, frame shape/pixel matching against ffmpeg output, and stream reading behavior with skip_frames.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A rabbit hops through rotated frames,
Where 90° turns reshape the view,
Metadata now knows the angle true,
Display-oriented pixels claim their names,
Tests prove each ffmpeg claim! 📹

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'fix(frames): apply display-matrix rotation when decoding frames' accurately and concisely describes the main change: applying display-matrix rotation to frames during decoding to fix vertical video orientation issues.
Docstring Coverage	✅ Passed	Docstring coverage is 94.44% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/display-matrix-rotation

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@simple_video_utils/frames.py`:
- Around line 236-246: In read_frames_from_stream the PyAV container created as
container = av.open(...) can leak if an exception occurs during setup (setting
s.thread_type, decoding first_frame, or computing meta via
video_metadata_from_container) because container.close() only lives in
frame_generator()'s finally; wrap the setup steps that touch container (the for
s in container.streams.video loop, first_frame = next(...), rotation assignment
and meta = video_metadata_from_container(...)) in a try/except or try/finally
that calls container.close() on error and re-raises, or use a context manager to
ensure container.close() is invoked if returning the generator fails, so the
container is always closed on setup-time failures.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e7e7e2cf-950e-4bb8-a7a9-eb7842205b19

📥 Commits

Reviewing files that changed from the base of the PR and between 2f111aa and 3474014.

⛔ Files ignored due to path filters (1)

• tests/assets/rotated90.mp4 is excluded by !**/*.mp4

📒 Files selected for processing (4)

• pyproject.toml
• simple_video_utils/frames.py
• simple_video_utils/metadata.py
• tests/test_rotation.py