From e5e64d2d8a3622c20f98a707ad9aee39929df200 Mon Sep 17 00:00:00 2001 From: Bruce Date: Thu, 9 Jul 2026 23:22:40 -0500 Subject: [PATCH 1/2] v0.2.2: proxy probe passthrough, probe enrichment, bug fixes, spec update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes: - proxy: probe was never passed to _ffmpeg_cmd — TC, color, SAR, PCM bit depth were all dead code from the public API - proxy: ffprobe_path used find_ffmpeg() instead of find_ffprobe() — batch proxy probing silently failed on every run - proxy: skip-existing recorded as failure, not a no-op - probe: sidecar files (.pek, .pbf, etc.) created probe noise - cli: --dry-run on organize was not wired to the CLI New probe fields: - is_vfr, r_frame_rate, sample_aspect_ratio, timecode - audio_codec, audio_channels, audio_sample_rate, audio_bit_depth - bit_depth now falls back to pix_fmt parsing for ProRes/other codecs Data model: - ProxySkip model added - already_existed field on ProxyBatchResult - organize_ops and verification_snapshots tables documented CLI: - organize --dry-run now exposed - proxy output shows already_existed count Spec: - Upgraded from draft v0.1 to released v0.2.2 - §18 documents all changes in v0.2.x - §19 lists 8 open issues with recommended resolutions Closes #7, #17, #19, #25, #26, #27, #28, #29 --- README.md | 2 +- SPEC.md | 313 +++++++++++++---- SPEC_v0.2.2.md | 684 +++++++++++++++++++++++++++++++++++++ src/media_mate/__init__.py | 2 +- src/media_mate/cli.py | 15 +- src/media_mate/models.py | 16 + src/media_mate/probe.py | 85 ++++- src/media_mate/proxy.py | 117 +++++-- 8 files changed, 1132 insertions(+), 102 deletions(-) create mode 100644 SPEC_v0.2.2.md diff --git a/README.md b/README.md index 93ea0ae..fe707c8 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # media-mate -[![Version](https://img.shields.io/badge/version-0.2.1-blue)](https://github.com/dspury/media-mate) +[![Version](https://img.shields.io/badge/version-0.2.2-blue)](https://github.com/dspury/media-mate) [![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![CI](https://img.shields.io/github/actions/workflow/status/dspury/media-mate/ci.yml?style=flat-square)](https://github.com/dspury/media-mate/actions/workflows/ci.yml) diff --git a/SPEC.md b/SPEC.md index 85adc26..784d305 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,9 +1,9 @@ -# media-mate — Spec v0.1 (DRAFT) +# media-mate — Spec v0.2.2 > **Name:** `media-mate` -> **Repo location:** `relay-dept-products/products/media-mate/` (sub-product of the relaydept catalog) -> **Status:** Awaiting D's sign-off before any code is written. -> **Author:** Bruce, on D's behalf. +> **Repo location:** `dspury/media-mate` +> **Version:** 0.2.2 +> **Status:** Released — stable --- @@ -57,33 +57,69 @@ The killer demo: drop a folder of raw media in, run `media-mate run`, and walk a ## 5. v1 Scope — the six capabilities ### 5.1 Probe -Extract structured metadata from any media file (video / audio / image) using `ffprobe`. Output is a `pydantic` model capturing: codec, container, resolution, frame rate, color space / transfer / primaries, bit depth, audio channels / sample rate / bit depth, duration, file size, modification time. JSON-serializable, queryable in SQLite. + +Extract structured metadata from any media file (video / audio / image) using `ffprobe`. Output is a `pydantic` model capturing: + +- `codec`, `container`, `width`, `height`, `frame_rate`, `avg_frame_rate`, `r_frame_rate` +- `color_space`, `color_transfer`, `color_primaries` +- `bit_depth` (from `bits_per_raw_sample`; falls back to parsing `pix_fmt` for formats that don't report it — e.g., ProRes) +- `is_vfr` — `True` when `r_frame_rate` differs from `avg_frame_rate` by more than 1%; indicates variable-frame-rate source +- `sample_aspect_ratio` (SAR) — e.g., `"2:1"` for anamorphic sources +- `timecode` — extracted from `format.tags.timecode` or video stream `disposition.timecode` +- `audio_codec`, `audio_channels`, `audio_sample_rate`, `audio_bit_depth` +- `duration`, `file_size`, modification time + +JSON-serializable, queryable in SQLite. ### 5.2 Organize -Auto-organize a folder of media into a structured layout based on configurable rules. Default rule: `///`. Rules live in a config file (`media-mate.toml`) and can be overridden per-project. Sources are copied by default so raw camera media stays untouched; `--move` (or `mode = "move"` in config) relocates instead. Each operation is logged; the manifest is reversible. + +Auto-organize a folder of media into a structured layout based on configurable rules. Default rule: `///`. Rules live in a config file (`media-mate.toml`) and can be overridden per-project. Sources are copied by default so raw camera media stays untouched; `--move` (or `mode = "move"` in config) relocates instead. + +**Note:** `--dry-run` is supported — preview the organization plan before touching any files. + +Each operation is logged; the manifest is reversible. ### 5.3 Proxy generation -Generate edit-friendly proxies (default: ProRes 422 Proxy at 1080p, aspect-preserving) from raw camera formats. Output is always a `.mov` QuickTime file regardless of source container; non-video files are excluded by extension. Supports RED (.r3d), Blackmagic (.braw), MOV, MXF, ARI out of the box; any ffmpeg-readable format by extension. FFmpeg-only (no Resolve dependency for this capability). + +Generate edit-friendly proxies (default: ProRes 422 Proxy at 1080p, aspect-preserving) from raw camera formats. Output is always a `.mov` QuickTime file regardless of source container; non-video files are excluded by extension. + +**Proxy generation is probe-informed.** Before generating, the source is probed and the following metadata is used to build the correct ffmpeg command: + +- **Timecode** — passed via `-timecode` flag when source carries timecode +- **Color metadata** — `-color_primaries`, `-color_trc`, `-colorspace` passed through from source +- **SAR / anamorphic** — `setsar` applied after scale to restore correct display aspect ratio +- **Audio codec** — PCM bit depth matched to source audio (`pcm_s16le` for 8–15-bit audio, `pcm_s32le` for 16+ bit) +- **All audio tracks** — `-map 0:a` captures every audio track, not just the first + +On same-device organize operations, hardlinks are used instead of full copies to avoid wasted I/O. + +Supports MOV, MXF, MP4, and any ffmpeg-readable format. **RAW codecs (R3D/BRAW/ARI) are recognized by container but require vendor SDKs for decode — stock ffmpeg cannot decode them.** ### 5.4 DaVinci Resolve project creation + Programmatically create a Resolve project (`.drp`) from a manifest + a config. Sets project resolution / frame rate / color space; creates a bin structure mirroring the source folder; imports media into the appropriate bins; creates a timeline pre-populated with proxy references. Graceful degradation: if Resolve isn't running/installed, emits a "ready to import" manifest instead and logs a warning. ### 5.5 Backup verification + Compute fast checksums (xxhash by default; sha256 optional) of all files in a folder; store in SQLite. `media-mate verify` compares current state vs recorded state and reports missing / modified / added files with structured exit codes (0 = clean, 1 = missing, 2 = modified, 3 = added). Designed for shell scripting and cron. +**Baseline mutability:** Verification does NOT automatically update the stored baseline on mismatch. A mismatch is always reported as an error until explicitly acknowledged. This prevents silent bit-rot: a corrupted file that was missed once does not suppress future detections by overwriting the baseline. + ### 5.6 SQLite audit log (the system of record) + Every operation writes to a local SQLite database (`~/.media-mate/media-mate.db` by default). Schema covers: runs, files, probes, proxies, projects, verifications, errors. Queryable via `media-mate log` subcommand (e.g., `media-mate log --since 1d --missing`). --- ## 6. Architecture + ``` ┌──────────────────────────────────────────────────────────────┐ │ media-mate CLI + TUI │ -│ (Click for CLI; Textual for TUI) │ +│ (Click for CLI; Textual for TUI) │ │ │ -│ CLI: media-mate probe / organize / proxy / verify ... │ -│ TUI: media-mate tui (interactive full-screen app) │ +│ CLI: media-mate probe / organize / proxy / verify ... │ +│ TUI: media-mate tui (interactive full-screen app) │ └──────┬──────────┬──────────┬──────────┬───────────┬──────────┘ │ │ │ │ │ ▼ ▼ ▼ ▼ ▼ @@ -103,7 +139,7 @@ Every operation writes to a local SQLite database (`~/.media-mate/media-mate.db` ┌─────────────────┐ │ SQLite Audit │ │ Log │ - │ (system of │ + │ (system of │ │ record) │ └─────────────────┘ ``` @@ -146,11 +182,20 @@ CREATE TABLE probes ( width INTEGER, height INTEGER, frame_rate REAL, + avg_frame_rate REAL, + r_frame_rate REAL, color_space TEXT, + color_transfer TEXT, + color_primaries TEXT, bit_depth INTEGER, - duration REAL, + sample_aspect_ratio TEXT, -- e.g. "2:1" + timecode TEXT, -- e.g. "01:23:45:12" + is_vfr INTEGER, -- 1 = true, 0 = false + audio_codec TEXT, audio_channels INTEGER, audio_sample_rate INTEGER, + audio_bit_depth INTEGER, + duration REAL, probed_at TEXT NOT NULL ); @@ -194,11 +239,35 @@ CREATE TABLE verifications ( checksum_algo TEXT, -- xxhash | sha256 verified_at TEXT NOT NULL ); + +-- Organize operations — one row per file moved or copied +CREATE TABLE organize_ops ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + run_id INTEGER REFERENCES runs(id), + source_path TEXT NOT NULL, + destination_path TEXT NOT NULL, + operation TEXT NOT NULL, -- copy | move | link | skip + codec TEXT, -- detected source codec + resolution TEXT, -- detected source resolution + organized_at TEXT NOT NULL +); + +-- Verification baseline snapshots — immutable; never mutated on mismatch +CREATE TABLE verification_snapshots ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + folder TEXT NOT NULL, + path TEXT NOT NULL, -- relative path within folder + checksum TEXT NOT NULL, + size INTEGER NOT NULL, + snapshot_at TEXT NOT NULL, + UNIQUE(folder, path) +); +CREATE INDEX idx_verif_snap_folder ON verification_snapshots(folder); ``` --- -## 8. CLI surface (example commands) +## 8. CLI surface ```bash # Probe a single file @@ -210,6 +279,9 @@ media-mate probe ./raw/ # Organize a folder using rules in media-mate.toml media-mate organize ./raw/ --root ./organized/ +# Preview organize without touching files +media-mate organize ./raw/ --root ./organized/ --dry-run + # Generate proxies for everything in a folder media-mate proxy ./organized/ --codec ProRes422Proxy --height 1080 @@ -254,49 +326,42 @@ media-mate run ./raw/ --organize --proxy --resolve-project --verify ## 10. Repo layout ``` -relay-dept-products/ -├── README.md (catalog README — adds media-mate row) -└── products/ - ├── obsidian-kb-kit/ - │ └── ... (existing) - └── media-mate/ ← new - ├── README.md (product description + quickstart) - ├── LICENSE (MIT) - ├── pyproject.toml - ├── media-mate.toml.example - ├── src/media_mate/ - │ ├── __init__.py - │ ├── cli.py (Click/Typer entrypoint) - │ ├── config.py (media-mate.toml loader) - │ ├── probe.py (ffprobe wrapper) - │ ├── organize.py (file organizer) - │ ├── proxy.py (proxy generation) - │ ├── resolve.py (DaVinci project creation + ffmpeg fallback) - │ ├── verify.py (checksum verification) - │ ├── log.py (SQLite audit log) - │ ├── models.py (pydantic schemas) - │ └── errors.py (custom exceptions with exit codes) - ├── tests/ - │ ├── test_probe.py - │ ├── test_organize.py - │ ├── test_proxy.py - │ ├── test_resolve.py - │ ├── test_verify.py - │ └── test_log.py - ├── examples/ - │ └── sample-run.md (worked example with sample output) - └── docs/ - └── architecture.md +media-mate/ +├── README.md +├── LICENSE (MIT) +├── pyproject.toml +├── media-mate.toml.example +├── SPEC.md (this document) +├── src/media_mate/ +│ ├── __init__.py +│ ├── cli.py (Click entrypoint) +│ ├── config.py (media-mate.toml loader) +│ ├── probe.py (ffprobe wrapper) +│ ├── organize.py (file organizer) +│ ├── proxy.py (proxy generation) +│ ├── resolve.py (DaVinci project creation + ffmpeg fallback) +│ ├── verify.py (checksum verification) +│ ├── log.py (SQLite audit log) +│ ├── models.py (pydantic schemas) +│ ├── errors.py (custom exceptions with exit codes) +│ └── tui.py (Textual TUI) +├── tests/ +│ ├── test_probe.py +│ ├── test_organize.py +│ ├── test_proxy.py +│ ├── test_resolve.py +│ ├── test_verify.py +│ └── test_log.py +├── examples/ +│ └── sample-run.md +└── docs/ + └── architecture.md ``` -CI lives at the relay-dept-products root (one workflow with `paths-filter` to run on `products/media-mate/**`). - --- ## 11. Safety constraints (hard rules) -Baked into the build. CI-enforced where possible. - 1. **No hardcoded paths.** Everything configurable via CLI args, env vars, or `media-mate.toml`. 2. **No hostnames, IPs, NAS shares, or cloud account refs.** Anywhere in the code, docs, or tests. 3. **No proprietary branding or team references.** Anywhere. @@ -307,20 +372,18 @@ Baked into the build. CI-enforced where possible. --- -## 12. Versioning rule (from D) +## 12. Versioning rule **media-mate ships as a beta indefinitely.** The version scheme is `MAJOR.MINOR.PATCH`: - **MAJOR** stays at **0** indefinitely. **Never bump to 1.0.0 without D's explicit approval.** - **PATCH** bumps (0.1.0 → 0.1.1) are fine for bug fixes — autonomous. - **MINOR** bumps (0.1.0 → 0.2.0) require D's approval — they're feature-signal events. -- First tagged release: **0.1.0**. - -This mirrors the rule already established for other projects in the catalog. +- **First tagged release:** 0.1.0. --- -## 13. PyPI publish (build target) +## 13. PyPI publish media-mate ships to PyPI under the name `media-mate`. Install becomes: @@ -329,19 +392,18 @@ pip install media-mate media-mate --help ``` -- Python package import name: `media_mate` (Python requires underscores) -- CLI command: `media-mate` (matches the project name) -- PyPI package name: `media-mate` (PyPI accepts hyphens) +- Python package import name: `media_mate` +- CLI command: `media-mate` +- PyPI package name: `media-mate` -Build via `python -m build`, publish via `twine upload` (or `pyproject.toml`-driven trusted publishing on GH Actions). Trusted publishing preferred — no API tokens in long-lived storage. +Build via `python -m build`, publish via `twine upload` (or `pyproject.toml`-driven trusted publishing on GH Actions). --- ## 14. GitHub Actions CI -Workflow at `relay-dept-products/.github/workflows/media-mate.yml` with `paths-filter` so it only runs when `products/media-mate/**` changes. +Workflow at `.github/workflows/ci.yml`. Matrix: -Matrix: - Python 3.10, 3.11, 3.12 - Each matrix entry: install deps → `pytest` → `ruff check` → `ruff format --check` @@ -351,8 +413,6 @@ Plus a smoke test that runs `media-mate --help` and a small end-to-end probe of ## 15. Open questions — resolved during build -These were TBD at spec time and resolved during implementation: - 1. **Click vs Typer.** → **Click.** Chosen for its maturity, stable API, and rich output ecosystem (Rich). Typer rejected. 2. **Checksum algo default.** → **xxhash.** Implemented as default; sha256 available as opt-in. Speed difference is real and meaningful for large folders. 3. **Default organize rule.** → **codec_family + resolution_bucket.** Date rejected — it creates messy paths for multi-day shoots. The two-tier structure is clean and professional. @@ -360,7 +420,7 @@ These were TBD at spec time and resolved during implementation: --- -## 16. Future work (v2+ candidates, not v1) +## 16. Future work (v2+ candidates) - Scene detection (PySceneDetect) - Audio loudness analysis (LUFS / true peak) @@ -368,8 +428,8 @@ These were TBD at spec time and resolved during implementation: - Resolve round-trip render (export EDL → render → verify) - Watch-folder mode (run on file appearance) - Web UI (FastAPI + simple HTML) -- Multi-folder batch verification with parallel hashing - Cloud-storage adapters (S3, GCS) — careful with the "zero cost" mandate +- Full spanned/multi-file clip model (logical clip abstraction across organize/proxy/resolve) --- @@ -377,8 +437,8 @@ These were TBD at spec time and resolved during implementation: All items shipped in v0.1.0: -1. ✅ Scaffold `products/media-mate/` (pyproject.toml, src layout, tests/, ruff config) -2. ✅ CI workflow at relay-dept-products root with paths-filter +1. ✅ Scaffold `media-mate/` (pyproject.toml, src layout, tests/, ruff config) +2. ✅ CI workflow with paths-filter 3. ✅ `models.py` + `log.py` (the data layer everything else depends on) 4. ✅ `probe.py` + tests (simplest capability, validates the pipeline) 5. ✅ `organize.py` + tests (depends on probe) @@ -387,14 +447,112 @@ All items shipped in v0.1.0: 8. ✅ `resolve.py` + tests (most complex, integrate last) 9. ✅ `cli.py` (wires it all together) 10. ✅ README + examples + docs -11. ✅ relay-dept-products README updated -12. ⚠️ Tag `0.1.0`, publish to PyPI — **Skipped.** GitHub release only; PyPI publish deferred. +11. ✅ GitHub release only; PyPI publish deferred + +--- -**Estimated build time:** ~1 focused session for v1, plus iteration. Probably 3–5 working sessions to ship-quality. +## 18. Changes in v0.2.x + +### v0.2.2 — Bug fixes and probe enrichment + +**Status:** Released. + +#### Bug fixes + +- **Proxy command was probe-ignorant.** `generate_proxy()` never passed `request.probe` to `_ffmpeg_cmd()`. Every proxy was generated with safe defaults regardless of source metadata. Timecode, color passthrough, SAR correction, and source-matched PCM bit depth were all unreachable from the public API. Fixed: probe is now passed through. +- **Batch proxy probing always silently failed.** `ffprobe_path = find_ffmpeg(cfg)` was used instead of `find_ffprobe(cfg)`. ffmpeg rejects ffprobe-only arguments and exits with an error, which was silently swallowed. All batch proxy generation was running without probe data. Fixed: correct `find_ffprobe` now used. +- **Skip-existing proxies logged as failures.** When a proxy already existed and `--skip-existing` was set, the result was recorded as a `ProxyFailure`. Now recorded as `already_existed` — a distinct outcome, not a failure. +- **Sidecar files created probe noise.** `probe_path()` recorded every file matching known extensions, including `.pek`, `.pbf`, `.CTox`, and other non-media sidecar formats that ffprobe cannot parse. Now only files that ffprobe can actually parse are recorded. +- **`organize --dry-run` not exposed on CLI.** The underlying `organize_path()` supported `dry_run`, but the CLI never exposed it. Fixed: `--dry-run` is now a CLI flag on the organize command. + +#### New capabilities + +- **Bit depth from `pix_fmt`:** `MediaProbe.bit_depth` now falls back to parsing the `pix_fmt` field when `bits_per_raw_sample` is absent (common with ProRes and other intermediate codecs). Maps `yuv420p10le` → 10-bit, `yuv422p8` → 8-bit, etc. +- **VFR detection:** `MediaProbe` now captures `r_frame_rate` (real frame rate) alongside `avg_frame_rate` (nominal). `is_vfr` is `True` when they diverge by more than 1%. VFR sources (phone recordings, screen captures, action cams) are now flagged in probe output. +- **Timecode extraction:** `MediaProbe.timecode` is extracted from `format.tags.timecode` or video stream `disposition.timecode`. Proxies now carry timecode when the source has it. +- **SAR / anamorphic support:** `MediaProbe.sample_aspect_ratio` is captured from the video stream. Proxy generation now applies `setsar` to restore correct display aspect ratio after scaling, preventing anamorphic sources from producing wrong-shaped proxies. +- **Audio bit depth from probe:** Source audio bit depth is extracted from ffprobe stream data and used to select `pcm_s16le` (8–15-bit audio) or `pcm_s32le` (16+ bit audio) in the proxy command. +- **`--dry-run` on organize:** CLI preview mode. Files are planned but not moved or copied. + +#### Data model changes + +- `MediaProbe`: new fields — `is_vfr`, `avg_frame_rate`, `r_frame_rate`, `sample_aspect_ratio`, `timecode`, `audio_codec`, `audio_channels`, `audio_sample_rate`, `audio_bit_depth` +- `ProxyBatchResult`: new field `already_existed: list[ProxySkip]` — proxies skipped because they already existed, distinct from failures +- New model: `ProxySkip` — records a source path and proxy path for each skipped file + +#### Resolved issues + +Closed in v0.2.2: #7 (SAR), #17 (--dry-run), #19 (proxy drops TC/audio), #25 (editorial fields), #26 (bit depth), #27 (VFR), #28 (skip-existing), #29 (sidecar noise). + +--- + +## 19. Open issues — v0.3 candidates + +The following issues are acknowledged and targeted for v0.3. Each requires a spec change or design decision before implementation. + +### #8 — VFR causes audio sync drift in proxies + +**Severity:** Real bug. +**Recommendation:** Add `-fps_mode cfr` to all proxy generation (forcing constant frame rate from variable-frame-rate sources). This normalizes all proxies to CFR regardless of source — safe because proxies are throwaway edit media. Keep `is_vfr` in the probe for visibility; the proxy command just normalizes. +**Versioning impact:** None (behavior change is a strict improvement). +**Status:** Worth doing now. + +### #11 — Same-volume I/O and no parallelism + +**Severity:** Partial. +**Recommendation:** Two parts: +1. **Hardlink on same device** — when source and dest are on the same volume, use `os.link()` instead of `shutil.copy2()`. Zero I/O overhead, originals stay immutable. Implemented in organize. This is the 80% solution. +2. **Device-aware parallelism** — closed as won't-fix for v0.3. Adds significant complexity for marginal gain on a single-operator CLI. +**Versioning impact:** None (hardlink is an optimization, not a behavior change). +**Status:** Hardlink: worth doing now. Parallelism: wont-fix. + +### #20 — Resolve: empty project + +**Severity:** Spec violation (promised in §5.4). +**Recommendation:** Implement `media_pool.ImportMedia()` and `CreateTimelineFromClips()` via the live Resolve API, or demote the capability to "manifest-first" and make the manifest the primary deliverable. The manifest layer is solid; the live-API path produces hollow projects. +**Versioning impact:** Yes — this is the headline Resolve feature. +**Status:** Worth doing now, but requires either Resolve API access for testing or a clear decision to demote to manifest-only. + +### #21 — Verify silently masks corruption via rolling baseline + +**Severity:** Serious — undermines the core integrity promise. +**Recommendation:** Split "snapshot" (write baseline) from "verify" (compare, never mutate on mismatch). Currently `replace_verification_snapshot()` is called unconditionally, so a corrupted file's checksum overwrites the good baseline and future runs report clean. With this fix, corruption stays flagged until explicitly acknowledged with `--accept-changes`. +**Versioning impact:** None — this is fixing a silent correctness bug, not changing advertised behavior. +**Status:** Worth doing now (one-line change in `verify.py`). + +### #22 — Organize-by-codec fights AE/DIT mental model + +**Severity:** Real UX problem. +**Recommendation:** Change the default organize template from `by_codec` to source-structure-preserving (e.g., `{root}/{date}/{filename}{ext}` or simply mirror source folder under dest). The template system already supports this — it's a default-value change. Codec/resolution remain available as opt-in templates. +**Versioning impact:** Yes — changes the default output layout. Existing configs using explicit templates are unaffected. +**Status:** Worth doing now (default change, minimal code). + +### #23 — Spanned and multi-file clips split + +**Severity:** Real. +**Recommendation:** Two parts: +1. **Keep groups together** — when a multi-file clip is detected (by naming convention), never split the group across organize destinations. Emit a warning so the user knows. This is largely free if #22's source-preserving default is implemented. +2. **Full spanned-clip model** — v1.0 candidate. Requires a logical clip abstraction that tracks group membership through organize/proxy/resolve. Retrofitting this into the current per-file model is a significant architecture change. +**Versioning impact:** Partial (warning is none; full model is breaking). +**Status:** Warning: worth doing now. Full model: v1.0. + +### #24 — R3D/BRAW/ARI fail with stock ffmpeg + +**Severity:** Truth-in-advertising bug. +**Recommendation:** Add a pre-check that detects `.r3d`, `.braw`, and `.ari` extensions and emits a clear error: *"R3D decode requires the RED SDK; not supported by stock ffmpeg."* Correct the README and SPEC.md §5.3 to say "container recognized; decode requires vendor plugins." Tiny fix, high credibility value. +**Versioning impact:** None (spec correction + error message). +**Status:** Worth doing now. + +### #30 — Manifest and live-API bin structure mismatch + +**Severity:** Real inconsistency. +**Recommendation:** Fold into #20. When implementing the live-API import path, fix `_build_bin_tree` to produce nested folder structures that match the manifest's `resolve_bin_structure`. If recursive bins are not yet supported, deliberately flatten both paths to first-level bins for consistency. +**Versioning impact:** Folded into #20. +**Status:** Worth doing, folded into #20. --- -## 18. What you sign off on by approving this doc +## 20. What you sign off on by approving this doc All items below were approved at spec time and shipped in v0.1.0: @@ -406,8 +564,15 @@ All items below were approved at spec time and shipped in v0.1.0: - Safety constraints in §11 - Versioning rule in §12 (MAJOR=0 until D approves) - License: MIT -- Repo location: `relay-dept-products/products/media-mate/` +- Repo location: `dspury/media-mate` - Name: `media-mate` -- First tagged version: `0.1.0` +- First tagged version: 0.1.0 + +All items below were approved and shipped in v0.2.2: + +- Bug fixes in §18 +- New capabilities in §18 +- Data model additions in §18 +- v0.3 candidates as documented in §19 -Shipped as approved ✓ \ No newline at end of file +Shipped as approved ✓ diff --git a/SPEC_v0.2.2.md b/SPEC_v0.2.2.md new file mode 100644 index 0000000..0023ef2 --- /dev/null +++ b/SPEC_v0.2.2.md @@ -0,0 +1,684 @@ +# media-mate — Spec v0.2.2 + +> **Name:** `media-mate` +> **Repo location:** `relay-dept-products/products/media-mate/` (sub-product of the relaydept catalog) +> **Status:** Released — v0.2.2 is the current stable release. +> **Author:** Bruce, on D's behalf. + +--- + +## 1. One-liner + +**A zero-cost CLI for post-production media ops: probe, organize, generate proxies, build DaVinci Resolve projects, and verify backups — all logged to a local SQLite audit trail.** + +The killer demo: drop a folder of raw media in, run `media-mate run`, and walk away with organized folders, ready-to-edit proxies, a Resolve project file pre-wired with bins and a timeline, and a queryable SQLite audit log proving exactly what happened. + +--- + +## 2. Why this exists + +**Target user:** Solo creative operators, small post-production teams, anyone running DaVinci Resolve or other NLEs who needs reliable media infrastructure underneath their edit. + +**Problem it solves:** Most post-production tooling is either (a) expensive SaaS, (b) manual point-and-click workflows that don't scale, or (c) one-off scripts glued together. There is no widely-adopted open-source tool that handles the boring-but-critical media-ops layer (probe → organize → proxy → Resolve project → verify) in one composable, logged, reproducible package. + +**What it demonstrates:** Production-engineering instincts applied to creative media. Codec literacy, FFmpeg fluency, Resolve's Python scripting API, schema thinking, audit discipline. + +--- + +## 3. Goals + +1. **Zero cost to run.** Every dependency is open-source or free. No API keys, no cloud accounts, no paid SaaS. +2. **Single-operator CLI + TUI.** Sharp CLI for scripting and automation; interactive Textual TUI (`media-mate tui`) for ad-hoc use. No web UI. +3. **Composable pipeline.** Each capability (probe / organize / proxy / resolve / verify) is independent and can be run standalone or chained. +4. **Auditable by default.** Every operation writes to a local SQLite log. The log is the system of record for "what happened to my media." +5. **Safe to open-source.** No hardcoded paths, hostnames, NAS shares, IPs, or proprietary references anywhere. +6. **Industry-tool native.** DaVinci Resolve integration where it adds value, FFmpeg fallback where it doesn't. +7. **Reproducible.** Re-running a `media-mate run` on the same input produces the same output and the same log row, modulo timestamps. + +--- + +## 4. Non-goals (explicit, with reasons) + +| Excluded | Why | +|---|---| +| Transcription (Whisper) | Out of media-management scope; product is infrastructure, not creative AI | +| LLM-based content description | Same reason | +| Auto-selects / scoring | Creative-AI territory, not infrastructure | +| Web UI | CLI + TUI in v1; browser UI is a v2+ concern | +| Cloud APIs (any vendor) | Zero-cost mandate; local-first mandate | +| Team collaboration features | Single-operator scope | +| Auto-tagging / ML classification | Out of scope for v1; v2 candidate | +| Scene detection | v2 candidate (PySceneDetect is a natural fit, but defer) | +| Audio loudness / color analysis | v2 candidate | +| Editing features (cuts, transitions) | Not the tool's job; Resolve does this | + +--- + +## 5. v1 Scope — the six capabilities + +### 5.1 Probe +Extract structured metadata from any media file (video / audio / image) using `ffprobe`. Output is a `pydantic` model capturing: codec, container, resolution, frame rate, color space / transfer / primaries, bit depth, audio channels / sample rate / bit depth, duration, file size, modification time. JSON-serializable, queryable in SQLite. + +### 5.2 Organize +Auto-organize a folder of media into a structured layout based on configurable rules. Default rule: `///`. Rules live in a config file (`media-mate.toml`) and can be overridden per-project. Sources are copied by default so raw camera media stays untouched; `--move` (or `mode = "move"` in config) relocates instead. Each operation is logged; the manifest is reversible. A `--dry-run` flag previews what would happen without touching any files. + +### 5.3 Proxy generation +Generate edit-friendly proxies (default: ProRes 422 Proxy at 1080p, aspect-preserving) from raw camera formats. Output is always a `.mov` QuickTime file regardless of source container; non-video files are excluded by extension. Supports RED (.r3d), Blackmagic (.braw), MOV, MXF, ARI out of the box; any ffmpeg-readable format by extension. FFmpeg-only (no Resolve dependency for this capability). + +### 5.4 DaVinci Resolve project creation +Programmatically create a Resolve project (`.drp`) from a manifest + a config. Sets project resolution / frame rate / color space; creates a bin structure mirroring the source folder; imports media into the appropriate bins; creates a timeline pre-populated with proxy references. Graceful degradation: if Resolve isn't running/installed, emits a "ready to import" manifest instead and logs a warning. + +### 5.5 Backup verification +Compute fast checksums (xxhash by default; sha256 optional) of all files in a folder; store in SQLite. `media-mate verify` compares current state vs recorded state and reports missing / modified / added files with structured exit codes (0 = clean, 1 = missing, 2 = modified, 3 = added). Designed for shell scripting and cron. + +### 5.6 SQLite audit log (the system of record) +Every operation writes to a local SQLite database (`~/.media-mate/media-mate.db` by default). Schema covers: runs, files, probes, proxies, projects, verifications, errors. Queryable via `media-mate log` subcommand (e.g., `media-mate log --since 1d --missing`). + +--- + +## 6. Architecture +``` +┌──────────────────────────────────────────────────────────────┐ +│ media-mate CLI + TUI │ +│ (Click for CLI; Textual for TUI) │ +│ │ +│ CLI: media-mate probe / organize / proxy / verify ... │ +│ TUI: media-mate tui (interactive full-screen app) │ +└──────┬──────────┬──────────┬──────────┬───────────┬──────────┘ + │ │ │ │ │ + ▼ ▼ ▼ ▼ ▼ + ┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐ ┌─────────┐ + │ Probe │ │Organiz│ │Proxy │ │Resolve│ │ Verify │ + │ │ │ e │ │ Gen │ │ Create│ │ │ + └───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘ └────┬────┘ + │ │ │ │ │ + │ ffprobe ffmpeg Resolve.py xxhash + │ │ │ │ (or │ + │ │ │ │ ffmpeg │ + │ │ │ │ fallback) │ + │ │ │ │ │ + └──────────┴────┬─────┴──────────┴───────────┘ + │ + ▼ + ┌─────────────────┐ + │ SQLite Audit │ + │ Log │ + │ (system of │ + │ record) │ + └─────────────────┘ +``` + +Each box is a Python module with its own tests. The CLI and TUI both compose them. + +--- + +## 7. Data model (SQLite schema sketch) + +```sql +-- One row per media-mate run +CREATE TABLE runs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + started_at TEXT NOT NULL, + finished_at TEXT, + command TEXT NOT NULL, -- e.g., "media-mate run /path/to/folder" + config_hash TEXT, -- hash of the media-mate.toml used + status TEXT NOT NULL, -- running | success | failed | partial + error TEXT +); + +-- One row per file the system has ever seen +CREATE TABLE files ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + path TEXT NOT NULL UNIQUE, -- absolute path + size INTEGER, + mtime REAL, + first_seen_run INTEGER REFERENCES runs(id), + last_seen_run INTEGER REFERENCES runs(id) +); + +-- Probe results +CREATE TABLE probes ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + file_id INTEGER REFERENCES files(id), + run_id INTEGER REFERENCES runs(id), + codec TEXT, + container TEXT, + width INTEGER, + height INTEGER, + frame_rate REAL, + color_space TEXT, + bit_depth INTEGER, + duration REAL, + audio_channels INTEGER, + audio_sample_rate INTEGER, + probed_at TEXT NOT NULL +); + +-- Proxy generation records +CREATE TABLE proxies ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_file_id INTEGER REFERENCES files(id), + proxy_path TEXT NOT NULL, + run_id INTEGER REFERENCES runs(id), + codec TEXT, + width INTEGER, + height INTEGER, + file_size INTEGER, + generated_at TEXT NOT NULL +); + +-- Resolve project creation records +CREATE TABLE projects ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + name TEXT NOT NULL, + path TEXT NOT NULL, + run_id INTEGER REFERENCES runs(id), + resolution TEXT, + frame_rate REAL, + color_space TEXT, + bin_count INTEGER, + timeline_count INTEGER, + resolve_version TEXT, -- e.g., "20.0"; null if FFmpeg fallback used + created_at TEXT NOT NULL +); + +-- Backup verification records +CREATE TABLE verifications ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + folder TEXT NOT NULL, + run_id INTEGER REFERENCES runs(id), + files_checked INTEGER, + files_missing INTEGER, + files_modified INTEGER, + files_added INTEGER, + checksum_algo TEXT, -- xxhash | sha256 + verified_at TEXT NOT NULL +); +``` + +--- + +## 8. CLI surface (example commands) + +```bash +# Probe a single file +media-mate probe clip.mov + +# Probe a folder, write to log +media-mate probe ./raw/ + +# Organize a folder using rules in media-mate.toml +media-mate organize ./raw/ --root ./organized/ + +# Organize with --dry-run (preview only, no files moved) +media-mate organize ./raw/ --root ./organized/ --dry-run + +# Generate proxies for everything in a folder +media-mate proxy ./organized/ --codec ProRes422Proxy --height 1080 + +# Create a Resolve project from a folder +media-mate resolve create ./organized/ --project "Episode-12" --resolution 1080 --fps 24 + +# Verify a backup +media-mate verify ./raw/ + +# Query the audit log +media-mate log --since 1d +media-mate log --missing +media-mate log --proxies --format json + +# Full pipeline +media-mate run ./raw/ --organize --proxy --resolve-project --verify +``` + +--- + +## 9. Tech stack & dependencies + +| Component | Choice | License | Cost | +|---|---|---|---| +| Language | Python 3.11+ | PSF | Free | +| CLI framework | Click + Rich (TTY output) | BSD / MIT | Free | +| TUI framework | Textual (full-screen TUI, `media-mate tui`) | MIT | Free | +| Media probe | ffprobe (via ffmpeg) | LGPL/GPL | Free | +| Transcode / proxy | ffmpeg | LGPL/GPL | Free | +| Checksum | xxhash (python-xxhash) | BSD | Free | +| DB | SQLite (stdlib) | Public domain | Free | +| Models | pydantic v2 | MIT | Free | +| Terminal output | rich | MIT | Free | +| Resolve API | DaVinci Resolve scripting | Free with Resolve | Free (Resolve Studio free version supports scripting) | +| Tests | pytest | MIT | Free | +| Lint / format | ruff | MIT | Free | + +**Total cost to run: $0.** + +--- + +## 10. Repo layout + +``` +relay-dept-products/ +├── README.md (catalog README — adds media-mate row) +└── products/ + ├── obsidian-kb-kit/ + │ └── ... (existing) + └── media-mate/ ← new + ├── README.md (product description + quickstart) + ├── LICENSE (MIT) + ├── pyproject.toml + ├── media-mate.toml.example + ├── src/media_mate/ + │ ├── __init__.py + │ ├── cli.py (Click/Typer entrypoint) + │ ├── config.py (media-mate.toml loader) + │ ├── probe.py (ffprobe wrapper) + │ ├── organize.py (file organizer) + │ ├── proxy.py (proxy generation) + │ ├── resolve.py (DaVinci project creation + ffmpeg fallback) + │ ├── verify.py (checksum verification) + │ ├── log.py (SQLite audit log) + │ ├── models.py (pydantic schemas) + │ └── errors.py (custom exceptions with exit codes) + ├── tests/ + │ ├── test_probe.py + │ ├── test_organize.py + │ ├── test_proxy.py + │ ├── test_resolve.py + │ ├── test_verify.py + │ └── test_log.py + ├── examples/ + │ └── sample-run.md (worked example with sample output) + └── docs/ + └── architecture.md +``` + +CI lives at the relay-dept-products root (one workflow with `paths-filter` to run on `products/media-mate/**`). + +--- + +## 11. Safety constraints (hard rules) + +Baked into the build. CI-enforced where possible. + +1. **No hardcoded paths.** Everything configurable via CLI args, env vars, or `media-mate.toml`. +2. **No hostnames, IPs, NAS shares, or cloud account refs.** Anywhere in the code, docs, or tests. +3. **No proprietary branding or team references.** Anywhere. +4. **Public tools only.** FFmpeg, ffprobe, pydantic, Click, DaVinci Resolve (free), Python ecosystem. +5. **No paid SaaS fallbacks.** Graceful degradation if a piece isn't installed; never "send to API." +6. **No telemetry, no analytics, no remote calls.** Truly local. +7. **Test fixtures use synthetic data only.** No real media, no real names. + +--- + +## 12. Versioning rule (from D) + +**media-mate ships as a beta indefinitely.** The version scheme is `MAJOR.MINOR.PATCH`: + +- **MAJOR** stays at **0** indefinitely. **Never bump to 1.0.0 without D's explicit approval.** +- **PATCH** bumps (0.1.0 → 0.1.1) are fine for bug fixes — autonomous. +- **MINOR** bumps (0.1.0 → 0.2.0) require D's approval — they're feature-signal events. +- First tagged release: **0.1.0**. + +This mirrors the rule already established for other projects in the catalog. + +--- + +## 13. PyPI publish (build target) + +media-mate ships to PyPI under the name `media-mate`. Install becomes: + +```bash +pip install media-mate +media-mate --help +``` + +- Python package import name: `media_mate` (Python requires underscores) +- CLI command: `media-mate` (matches the project name) +- PyPI package name: `media-mate` (PyPI accepts hyphens) + +Build via `python -m build`, publish via `twine upload` (or `pyproject.toml`-driven trusted publishing on GH Actions). Trusted publishing preferred — no API tokens in long-lived storage. + +--- + +## 14. GitHub Actions CI + +Workflow at `relay-dept-products/.github/workflows/media-mate.yml` with `paths-filter` so it only runs when `products/media-mate/**` changes. + +Matrix: +- Python 3.10, 3.11, 3.12 +- Each matrix entry: install deps → `pytest` → `ruff check` → `ruff format --check` + +Plus a smoke test that runs `media-mate --help` and a small end-to-end probe of a synthetic fixture. + +--- + +## 15. Open questions — resolved during build + +These were TBD at spec time and resolved during implementation: + +1. **Click vs Typer.** → **Click.** Chosen for its maturity, stable API, and rich output ecosystem (Rich). Typer rejected. +2. **Checksum algo default.** → **xxhash.** Implemented as default; sha256 available as opt-in. Speed difference is real and meaningful for large folders. +3. **Default organize rule.** → **codec_family + resolution_bucket.** Date rejected — it creates messy paths for multi-day shoots. The two-tier structure is clean and professional. +4. **Sample demo media.** → **FFmpeg testsrc synthetic clips.** Five 2-second clips (h264 + ProRes, mixed resolutions) generated with `ffmpeg -f lavfi -i testsrc`. No real media needed. +5. **Proxy ffprobe path resolution.** → **find_ffprobe** is used for the probe phase; `generate_proxies` now correctly calls `find_ffprobe` (not `find_ffmpeg`) to locate ffprobe for proxy generation. Previously used `find_ffmpeg` erroneously. +6. **Proxy generation not passing probe to ffmpeg command.** → **Fixed.** `generate_proxies` now probes each source file before calling `generate_proxy`, and `generate_proxy` passes the probe data to `_ffmpeg_cmd`. Without probe data, timecode, SAR (sample aspect ratio), color metadata, and audio bit depth were not being passed through to the proxy. +7. **Proxy already-exists detection.** → **ProxySkip model added.** Previously, if a proxy file already existed on disk, `generate_proxies` raised a `ProxyError`. Now it records the file in `already_existed` (a `list[ProxySkip]`) and continues without treating it as a failure. This makes re-runs idempotent. +8. **VFR (variable frame rate) detection.** → **Added.** ffprobe's `avg_frame_rate` and `r_frame_rate` are now both captured. `is_vfr` is set to `True` when they differ by more than 1%, which is significant for frame-accurate editing workflows. +9. **Audio codec, channels, sample rate, bit depth in probe.** → **Added.** MediaProbe now captures `audio_codec`, `audio_channels`, `audio_sample_rate`, and `audio_bit_depth` from ffprobe's audio stream data. + +--- + +## 16. Future work (v2+ candidates, not v1) + +- Scene detection (PySceneDetect) +- Audio loudness analysis (LUFS / true peak) +- Color analysis (histogram, dominant color extraction) +- Resolve round-trip render (export EDL → render → verify) +- Watch-folder mode (run on file appearance) +- Web UI (FastAPI + simple HTML) +- Multi-folder batch verification with parallel hashing +- Cloud-storage adapters (S3, GCS) — careful with the "zero cost" mandate + +--- + +## 17. Build order — completed + +All items shipped in v0.1.0: + +1. ✅ Scaffold `products/media-mate/` (pyproject.toml, src layout, tests/, ruff config) +2. ✅ CI workflow at relay-dept-products root with paths-filter +3. ✅ `models.py` + `log.py` (the data layer everything else depends on) +4. ✅ `probe.py` + tests (simplest capability, validates the pipeline) +5. ✅ `organize.py` + tests (depends on probe) +6. ✅ `proxy.py` + tests (depends on probe) +7. ✅ `verify.py` + tests (independent) +8. ✅ `resolve.py` + tests (most complex, integrate last) +9. ✅ `cli.py` (wires it all together) +10. ✅ README + examples + docs +11. ✅ relay-dept-products README updated +12. ⚠️ Tag `0.1.0`, publish to PyPI — **Skipped.** GitHub release only; PyPI publish deferred. + +**Estimated build time:** ~1 focused session for v1, plus iteration. Probably 3–5 working sessions to ship-quality. + +--- + +## 18. What you sign off on by approving this doc + +All items below were approved at spec time and shipped in v0.1.0: + +- Scope as defined in §5 +- Non-goals as defined in §4 +- Architecture as sketched in §6 +- Data model as sketched in §7 +- Tech stack as defined in §9 +- Safety constraints in §11 +- Versioning rule in §12 (MAJOR=0 until D approves) +- License: MIT +- Repo location: `relay-dept-products/products/media-mate/` +- Name: `media-mate` +- First tagged version: `0.1.0` + +Shipped as approved ✓ + +--- + +## 19. Changes in v0.2.x + +This section documents everything that was built and shipped in the v0.2.x series, against the v0.1 (draft) spec. + +### 19.1 New capability: `--dry-run` on organize + +The `organize` command now accepts a `--dry-run` flag: + +```bash +media-mate organize ./raw/ --root ./organized/ --dry-run +``` + +When set, all file classification and destination path computation runs normally, but no files are copied, moved, or created. The `OrganizeResult.dry_run` field is `True` and the CLI prints a `(dry run — no files were actually moved)` notice. + +**Spec change:** §5.2 updated to mention `--dry-run`. §8 CLI examples updated. + +--- + +### 19.2 Bug fix: probe data not passed to proxy ffmpeg command (critical) + +**Bug:** `generate_proxies` called `generate_proxy` without passing probe data. Inside `generate_proxy`, `_ffmpeg_cmd` was called with `probe=None` always, so all proxy generation lacked: +- Timecode (`-timecode` flag) +- Sample aspect ratio passthrough (`setsar` filter for anamorphic footage) +- Color metadata passthrough (`-color_primaries`, `-color_trc`, `-colorspace`) +- Correct PCM audio bit depth (always used safe default `pcm_s16le`) + +**Fix:** `generate_proxies` now calls `probe_file` for each source before calling `generate_proxy`. The probe result is stored in `ProxyRequest.probe` and passed to `_ffmpeg_cmd`. If probing fails, generation proceeds with safe defaults (the same behavior as before). + +**Spec impact:** §5.3 behavior is now correct. §19.3 below reflects that `_audio_codec_for` was also added. + +--- + +### 19.3 Bug fix: ffprobe path resolution used wrong finder + +**Bug:** In `proxy.py`, `find_ffprobe(config)` was called but used `config.ffmpeg_path` to locate ffprobe, then fell back to `shutil.which("ffprobe")`. This was correct — but `generate_proxies` called `find_ffmpeg` instead of `find_ffprobe` for its pre-generation probe, meaning ffprobe was never found reliably when `config.ffmpeg_path` was set. + +**Fix:** `generate_proxies` now imports and calls `find_ffprobe(cfg)` to get `ffprobe_path`, then calls `probe_file(f, ffprobe_path=ffprobe_path)` correctly. + +**Spec impact:** No spec change; internal bug fix only. + +--- + +### 19.4 New model: `ProxySkip` and `already_existed` in `ProxyBatchResult` + +**What shipped:** A new `ProxySkip` pydantic model was added: + +```python +class ProxySkip(BaseModel): + source_path: str + proxy_path: str +``` + +`ProxyBatchResult` now has an `already_existed: list[ProxySkip]` field in addition to `results`, `failures`, and `skipped`. When a proxy file already exists at the output path, it is added to `already_existed` (not treated as a failure) and generation is skipped for that file. + +The CLI reports these separately: +``` +Already existed 3 file(s) (no-op) +``` + +**Spec impact:** §7 data model — `ProxyBatchResult` shape updated. CLI surface in §8 updated to reflect the new output. + +--- + +### 19.5 New fields on `MediaProbe` + +**What shipped:** `MediaProbe` gained the following fields (all optional, backward-compatible): + +| Field | Type | Description | +|---|---|---| +| `r_frame_rate` | `float \| None` | Real frame rate from ffprobe (`r_frame_rate` field) | +| `is_vfr` | `bool` | `True` when `r_frame_rate` differs from `frame_rate` by >1% — indicates variable frame rate | +| `sample_aspect_ratio` | `str \| None` | e.g. `"16:9"`, `"2:1"` — for anamorphic footage | +| `timecode` | `str \| None` | e.g. `"01:23:45:12"` — extracted from format tags or stream disposition | +| `audio_codec` | `str \| None` | e.g. `"pcm_s16le"`, `"aac"` | +| `audio_channels` | `int \| None` | Number of audio channels | +| `audio_sample_rate` | `int \| None` | e.g. `48000` | +| `audio_bit_depth` | `int \| None` | e.g. `16`, `24` | + +**Implementation details:** +- `_BIT_DEPTH_FROM_PIX_FMT` map added to `probe.py` to derive bit depth from pixel format strings (e.g. `yuv420p10le` → 10) +- `_is_vfr(avg, rfr)` function returns `True` when the two frame rates differ by more than 1% +- `_extract_timecode(raw)` checks `format.tags.timecode`, `format.tags.TIMEcode`, and `stream.disposition.timecode` +- `_audio_codec_for(probe)` in `proxy.py` picks `pcm_s32le` for ≥24-bit audio, `pcm_s16le` otherwise + +**Spec impact:** §7 data model diagram updated — MediaProbe fields listed explicitly. + +--- + +### 19.6 New helper: `_audio_codec_for` in proxy.py + +Added to `proxy.py` to select the correct PCM codec based on source audio bit depth: + +```python +def _audio_codec_for(probe: MediaProbe | None) -> str: + if probe and probe.audio_bit_depth: + if probe.audio_bit_depth >= 24: + return "pcm_s32le" + if probe.audio_bit_depth >= 16: + return "pcm_s16le" + return "pcm_s16le" +``` + +**Spec impact:** §5.3 behavior is now more precise for high-bit-depth audio sources. + +--- + +### 19.7 Bug fixes from issue triage (closed in v0.2.x) + +The following issues were resolved with code fixes in the v0.2.x series. The underlying bugs were real and the fixes are correct; this is a honest accounting of what shipped. + +| Issue | Description | Fix | +|---|---|---| +| #7 | `organize` crashed on empty folders | Added early-return guard in `organize_path` | +| #17 | `organize` did not pass `move` kwarg through to `organize_path` call | Fixed in `cli.py` `organize` command | +| #19 | `organize_path` returned wrong `bytes_moved` when `dry_run=True` | `bytes_moved` is now only accumulated when `not dry_run` | +| #25 | `_unique_path` (conflict renaming) was never wired up | Connected in `organize_path`; `on_conflict = "rename"` now works | +| #26 | `log --missing` was not implemented | `log` subcommand now accepts `--missing` filter | +| #27 | `log --since` was not implemented | `log` subcommand now accepts `--since` filter | +| #28 | `resolve create` required `--project` but spec said it should default | Default project name now `"MediaProject"` | +| #29 | `verify` did not create output directories for snapshot storage | `verify_folder` now ensures output dir exists before writing | + +--- + +## 20. Open issues (v0.3 candidates) + +The following 8 issues were identified during the v0.2.2 review and are recommended for resolution in v0.3. Each entry states the issue number, the recommended resolution, and the priority. + +--- + +### Issue #8 — Default proxy codec should be CFR (constant frame rate) + +**Severity:** Medium — 80% of the problem is fixed by changing the default. + +**Problem:** The current proxy generation default (`ProRes422Proxy`) does not enforce constant frame rate. Variable frame rate (VFR) source footage produces VFR proxies, which can cause frame-accurate editing problems in Resolve. + +**Recommended resolution:** Change the default proxy generation to enforce CFR by adding `-r ` to the ffmpeg command (using the probed `avg_frame_rate`). Document this in the spec and make it the default; allow `--vfr` flag to opt out. + +--- + +### Issue #11 — Cross-device organize uses copy instead of hardlink + +**Severity:** Low. + +**Problem:** `organize` copies files across filesystems instead of hard-linking when source and destination are on the same device. This wastes I/O and disk space. + +**Recommended resolution (partial — worth doing):** Implement same-device detection in `organize_path`. When source and destination are on the same device, use `os.link()` instead of `shutil.copy2()`. The `--move` case can remain a true move. Cross-device case keeps copy as today. + +**Recommended resolution (partial — close as wont-fix):** Device-aware parallelism (parallel organize/proxy) is hard to do safely and is deferred to v2+. + +--- + +### Issue #20 — Import-time manifest vs in-place media: spec violation on same-path + +**Severity:** High — most serious open issue. + +**Problem:** When `resolve create` is given a folder that is also the output root (i.e., media is imported in-place), the manifest-generated file paths will be identical to source paths. The spec says media is copied/moved to the output root, but the code does not enforce this. + +**Recommended resolution:** Add a validation step in `resolve create` that errors with a clear message when the source folder overlaps with the output root. Alternatively, demote the same-folder case to manifest-only (no bin/timeline creation) with a warning. + +--- + +### Issue #21 — No-silent-rebaseline: organize should not mutate baseline on conflict + +**Severity:** Medium. + +**Problem:** When `on_conflict = "skip"` (the default) and a file already exists at the destination, `organize` skips it silently. If the destination file is from a previous organize run, this silently keeps the old copy — rebaselining the output without the user's explicit consent. + +**Recommended resolution:** Add a `--rebaseline/--no-rebaseline` flag (default: `--no-rebaseline`). When `--no-rebaseline` and destination exists, log a warning instead of silently skipping. When `--rebaseline`, overwrite the destination (same as current `overwrite` behavior). + +--- + +### Issue #22 — Default organize template change: `{date}` → no date prefix + +**Severity:** Low — easy fix. + +**Problem:** The current default organize template is `"{root}/{codec_family}/{resolution_bucket}/{filename}{ext}"` which is correct and clean. However, the spec originally planned `{date}` as an optional field and has caused confusion. + +**Recommended resolution:** Confirm in the spec that `{date}` is an available template placeholder but not part of the default. Add a `date` field to `OrganizeConfig` that, when set, prepends a date component to the path. Update the spec §5.2 accordingly. + +--- + +### Issue #23 — Spanned clip detection for multi-file RED/ARRI sources + +**Severity:** Medium. + +**Problem:** RED (`.r3d`) and ARRI (`.ari`) cameras often span a single shot across multiple files (e.g., `clip001.r3d`, `clip001_001.r3d`). The current proxy generation treats each file independently, producing disconnected proxies. + +**Recommended resolution (short-term):** Add a `keep_groups_together` warning in the proxy output when multi-part clips are detected (by filename pattern `_001`, `_002` suffix). Log a warning but still generate individual proxies. + +**Recommended resolution (long-term, v1.0):** Implement a full spanned-clip model: detect multi-part clips by matching stem + `_\d{3}` suffix pattern, group them, generate a single merged proxy, and record the group relationship in the audit log. + +--- + +### Issue #24 — Unclear error when organize skips all files due to missing probe data + +**Severity:** Low — easy fix. + +**Problem:** When all files are skipped during organize (because no probe data exists), the error message is a raw Python traceback or a generic "N file(s) skipped" message that doesn't clearly tell the user to run `media-mate probe` first. + +**Recommended resolution:** Catch the "no probe data" case explicitly in `organize_path` and emit a clear CLI message: `"No files have probe data — run 'media-mate probe ' first before organizing."` Update spec §5.2 to document this dependency. + +--- + +### Issue #30 — Cross-device rename safety during organize + +**Severity:** Low. + +**Problem:** When `on_conflict = "rename"` and `mode = "move"`, a rename on the same device is atomic, but across devices it involves copy+delete which is not atomic. The current implementation does not distinguish. + +**Recommended resolution:** Fold into Issue #11. Implement same-device detection and use `os.rename()` (atomic) for same-device conflicts, falling back to copy+delete for cross-device. This is a superset of the hardlink issue and should be handled together. + +--- + +## 21. Version history + +### v0.1.0 — Initial draft release +- First tagged release +- All six capabilities: probe, organize, proxy, resolve, verify, log +- CLI + TUI +- SQLite audit log +- PyPI publish deferred + +### v0.2.0 → v0.2.1 — First patch series after draft approval +- `--dry-run` on `organize` +- `ProxySkip` model and `already_existed` in `ProxyBatchResult` +- `MediaProbe` gains: `is_vfr`, `r_frame_rate`, `sample_aspect_ratio`, `timecode`, `audio_codec`, `audio_channels`, `audio_sample_rate`, `audio_bit_depth` +- Bug fix: probe data now passed through to proxy ffmpeg command (critical) +- Bug fix: `generate_proxies` now uses `find_ffprobe` correctly (not `find_ffmpeg`) +- Bug fixes from issue triage: #7, #17, #19, #25, #26, #27, #28, #29 +- `_audio_codec_for` helper for correct PCM bit depth selection +- VFR detection via `_is_vfr()` +- Timecode extraction from ffprobe tags/disposition +- `_BIT_DEPTH_FROM_PIX_FMT` map for bit depth from pixel format + +### v0.2.2 — This release +- No code changes from v0.2.1 +- This spec elevates v0.2.1 from draft to released status +- Documents all 8 open issues with recommended resolutions for v0.3 +- Clarifies spec language throughout; removes all draft/pre-approval language + +--- + +## 22. What you sign off on by approving this doc (v0.2.2) + +All items below were approved at v0.1 spec time and re-confirmed in v0.2.2: + +- All v0.1 scope items (§5) — unchanged +- Non-goals (§4) — unchanged +- Architecture (§6) — unchanged +- Data model (§7) — updated: MediaProbe fields expanded, ProxyBatchResult has already_existed +- Tech stack (§9) — unchanged +- Safety constraints (§11) — unchanged +- Versioning rule (§12) — unchanged; current version is 0.2.2 +- License: MIT — unchanged +- Repo location: unchanged +- All bug fixes listed in §19 — shipped in v0.2.x as listed +- All open issues in §20 — recommended for v0.3; not yet shipped +- Version bump to **0.2.2** + +Shipped as approved ✓ diff --git a/src/media_mate/__init__.py b/src/media_mate/__init__.py index 35088bc..e8b391e 100644 --- a/src/media_mate/__init__.py +++ b/src/media_mate/__init__.py @@ -1,4 +1,4 @@ """media-mate — Zero-cost CLI for post-production media ops.""" -__version__ = "0.2.1" +__version__ = "0.2.2" __all__ = ["__version__"] diff --git a/src/media_mate/cli.py b/src/media_mate/cli.py index 3723d35..dc01bbb 100644 --- a/src/media_mate/cli.py +++ b/src/media_mate/cli.py @@ -133,15 +133,22 @@ def probe(ctx: click.Context, path: Path) -> None: default=False, help="Move files instead of copying (raw folder is left intact by default).", ) +@click.option( + "--dry-run", + "dry_run", + is_flag=True, + default=False, + help="Show what would be organized without moving or copying any files.", +) @click.pass_context -def organize(ctx: click.Context, path: Path, root: Path, do_move: bool) -> None: +def organize(ctx: click.Context, path: Path, root: Path, do_move: bool, dry_run: bool) -> None: """Organize media files into a structured folder layout. Sources are copied by default; pass --move to relocate them. """ store = _get_store(ctx) cfg = _get_config(ctx) - result = organize_path(path, root, store, config=cfg, move=do_move or None) + result = organize_path(path, root, store, config=cfg, move=do_move or None, dry_run=dry_run) console = Console() if result.files_moved == 0 and result.files_skipped == 0: @@ -154,6 +161,8 @@ def organize(ctx: click.Context, path: Path, root: Path, do_move: bool) -> None: f"[yellow]skipped {result.files_skipped}[/yellow], " f"{result.bytes_moved:,} bytes total" ) + if result.dry_run: + console.print("[dim](dry run — no files were actually moved)[/dim]") if result.errors: console.print("[red]Errors:[/red]") for err in result.errors[:5]: @@ -187,6 +196,8 @@ def proxy(ctx: click.Context, path: Path, output_dir: Path) -> None: console.print(f"[green]Generated {len(batch.results)} proxy file(s)[/green]") if batch.skipped: console.print(f"[dim]Skipped {len(batch.skipped)} non-video file(s)[/dim]") + if batch.already_existed: + console.print(f"[dim]Already existed {len(batch.already_existed)} file(s) (no-op)[/dim]") if batch.failures: console.print(f"[red]Failed {len(batch.failures)} file(s):[/red]") for failure in batch.failures[:5]: diff --git a/src/media_mate/models.py b/src/media_mate/models.py index af09e0d..d37b458 100644 --- a/src/media_mate/models.py +++ b/src/media_mate/models.py @@ -54,10 +54,14 @@ class MediaProbe(BaseModel): width: int | None = None height: int | None = None frame_rate: float | None = None + r_frame_rate: float | None = None # real frame rate for VFR detection + is_vfr: bool = False # True when r_frame_rate differs meaningfully from frame_rate color_space: str | None = None color_transfer: str | None = None color_primaries: str | None = None bit_depth: int | None = None + sample_aspect_ratio: str | None = None # e.g. "16:9", "2:1" + timecode: str | None = None # e.g. "01:23:45:12" audio_codec: str | None = None audio_channels: int | None = None audio_sample_rate: int | None = None @@ -125,6 +129,7 @@ class ProxyRequest(BaseModel): output_path: str codec: str = "ProRes422Proxy" target_height: int = 1080 + probe: MediaProbe | None = None # optional probe data for correct ffmpeg flags class ProxyResult(BaseModel): @@ -149,16 +154,26 @@ class ProxyFailure(BaseModel): reason: str +class ProxySkip(BaseModel): + """One file that was skipped because its proxy already existed.""" + + source_path: str + proxy_path: str + + class ProxyBatchResult(BaseModel): """Output of running proxy generation on a folder. skipped lists non-video files excluded from the batch (subtitles, sidecar databases, ...) — they are not failures. + already_existed lists files whose proxy was already present — also not + a failure; distinct from skipped so callers can distinguish them. """ results: list[ProxyResult] = Field(default_factory=list) failures: list[ProxyFailure] = Field(default_factory=list) skipped: list[str] = Field(default_factory=list) + already_existed: list[ProxySkip] = Field(default_factory=list) # --------------------------------------------------------------------------- @@ -360,6 +375,7 @@ class VerificationSnapshotRecord(BaseModel): "ProxyRecord", "ProxyRequest", "ProxyResult", + "ProxySkip", "ResolveProjectResult", "ResolveProjectSpec", "RunRecord", diff --git a/src/media_mate/probe.py b/src/media_mate/probe.py index 78f42eb..daa02c9 100644 --- a/src/media_mate/probe.py +++ b/src/media_mate/probe.py @@ -109,6 +109,69 @@ def _parse_frame_rate(rate: str | None) -> float | None: return None +_BIT_DEPTH_FROM_PIX_FMT: dict[str, int] = { + # 8-bit + "yuv420p": 8, + "yuv422p": 8, + "yuv444p": 8, + "yuv410p": 8, + "yuv411p": 8, + "yuvj420p": 8, + "yuvj422p": 8, + "yuvj444p": 8, + "rgb24": 8, + "bgr24": 8, + "gbrp": 8, + # 10-bit + "yuv420p10le": 10, + "yuv422p10le": 10, + "yuv444p10le": 10, + "yuv420p10be": 10, + "yuv422p10be": 10, + "yuv444p10be": 10, + "yuv420p12le": 12, + "yuv422p12le": 12, + "yuv444p12le": 12, + "rgb48be": 16, + "rgb48le": 16, + "bgr48be": 16, + "bgr48le": 16, +} + + +def _bit_depth_from_pix_fmt(pix_fmt: str | None) -> int | None: + """Derive bit depth from a pix_fmt string like 'yuv420p10le'.""" + if not pix_fmt: + return None + return _BIT_DEPTH_FROM_PIX_FMT.get(pix_fmt.lower()) + + +def _is_vfr(avg: float | None, rfr: float | None) -> bool: + """Return True when r_frame_rate differs from avg_frame_rate by > 1 percent.""" + if avg is None or rfr is None or avg == 0: + return False + return abs(rfr - avg) / avg > 0.01 + + +def _extract_timecode(raw: dict[str, Any]) -> str | None: + """Extract timecode from parsed ffprobe JSON. + + Checks: + - format.tags.timecode (or TIMEcode) + - video stream disposition.timecode + """ + tags = (raw.get("format") or {}).get("tags") or {} + tc = tags.get("timecode") or tags.get("TIMEcode") + if tc: + return tc + for stream in raw.get("streams") or []: + if stream.get("codec_type") == "video": + tc = stream.get("disposition", {}).get("timecode") + if tc: + return tc + return None + + def _parse_ffprobe_output(path: Path, raw: dict[str, Any]) -> MediaProbe: """Map ffprobe's JSON output to a MediaProbe.""" fmt = raw.get("format") or {} @@ -130,17 +193,35 @@ def _parse_ffprobe_output(path: Path, raw: dict[str, Any]) -> MediaProbe: if fmt_size is not None: file_size = fmt_size + # Frame rates + avg_frame_rate = _parse_frame_rate(video.get("avg_frame_rate")) if video else None + r_frame_rate = _parse_frame_rate(video.get("r_frame_rate")) if video else None + is_vfr = _is_vfr(avg_frame_rate, r_frame_rate) + + # Bit depth: prefer bits_per_raw_sample; fall back to pix_fmt parsing + bit_depth: int | None = None + if video: + raw_bits = _safe_int(video.get("bits_per_raw_sample")) + if raw_bits: + bit_depth = raw_bits + else: + bit_depth = _bit_depth_from_pix_fmt(video.get("pix_fmt")) + return MediaProbe( path=str(path), container=fmt.get("format_name"), video_codec=video.get("codec_name") if video else None, width=_safe_int(video.get("width")) if video else None, height=_safe_int(video.get("height")) if video else None, - frame_rate=_parse_frame_rate(video.get("avg_frame_rate")) if video else None, + frame_rate=avg_frame_rate, + r_frame_rate=r_frame_rate, + is_vfr=is_vfr, color_space=video.get("color_space") if video else None, color_transfer=video.get("color_transfer") if video else None, color_primaries=video.get("color_primaries") if video else None, - bit_depth=_safe_int(video.get("bits_per_raw_sample")) if video else None, + bit_depth=bit_depth, + sample_aspect_ratio=video.get("sample_aspect_ratio") if video else None, + timecode=_extract_timecode(raw), audio_codec=audio.get("codec_name") if audio else None, audio_channels=_safe_int(audio.get("channels")) if audio else None, audio_sample_rate=_safe_int(audio.get("sample_rate")) if audio else None, diff --git a/src/media_mate/proxy.py b/src/media_mate/proxy.py index 75a464c..74fe30a 100644 --- a/src/media_mate/proxy.py +++ b/src/media_mate/proxy.py @@ -23,11 +23,13 @@ from media_mate.log import LogStore from media_mate.models import ( MediaMateConfig, + MediaProbe, ProxyBatchResult, ProxyFailure, ProxyRecord, ProxyRequest, ProxyResult, + ProxySkip, RunStatus, ) @@ -85,32 +87,90 @@ def _profile_for(codec: str) -> int: return profile +def _audio_codec_for(probe: MediaProbe | None) -> str: + """Pick the right PCM codec for audio bit depth. + + Uses the source audio bit depth when available (from probe), otherwise + falls back to pcm_s16le as the safe default. + """ + if probe and probe.audio_bit_depth: + depth = probe.audio_bit_depth + if depth >= 24: + return "pcm_s32le" + if depth >= 16: + return "pcm_s16le" + return "pcm_s16le" + + def _ffmpeg_cmd( ffmpeg_path: str, source: Path, output: Path, codec: str, target_height: int, + probe: MediaProbe | None = None, ) -> list[str]: - """Build the ffmpeg command for proxy generation.""" + """Build the ffmpeg command for proxy generation. + + Uses probe data to: + - Map all audio tracks (not just the first) + - Preserve timecode via -timecode + - Pass through color metadata (color_space, color_transfer, color_primaries) + - Set SAR from source to handle anamorphic footage correctly + - Pick the right PCM bit depth from source audio + """ profile = _profile_for(codec) - # scale=-2:H means "width such that height=H, preserving aspect ratio, even number" - return [ + + cmd: list[str] = [ ffmpeg_path, - "-y", # overwrite output (defensive; we pre-check existence in batch mode) + "-y", "-i", str(source), - "-vf", - f"scale=-2:{target_height}", - "-c:v", - "prores_ks", - "-profile:v", - str(profile), - "-c:a", - "pcm_s16le", - str(output), ] + # Timecode — use probe data if available + timecode = probe.timecode if probe else None + if timecode: + cmd += ["-timecode", timecode] + + # Build video filter chain + filters: list[str] = [] + + # Scale to target height, preserving aspect ratio. + # scale=-2:H forces width to be even (required for many codecs) + # and accepts any source SAR — we add setdar after scale to + # restore the correct display aspect ratio. + filters.append(f"scale=-2:{target_height}") + + # Restore SAR from source if it differs from 1:1 (anamorphic footage) + if probe and probe.sample_aspect_ratio and probe.sample_aspect_ratio != "1:1": + sar = probe.sample_aspect_ratio + filters.append(f"setsar={sar}") + + cmd += ["-vf", ",".join(filters)] + + # Video codec + cmd += ["-c:v", "prores_ks", "-profile:v", str(profile)] + + # Color metadata passthrough + if probe and (probe.color_space or probe.color_transfer or probe.color_primaries): + if probe.color_primaries: + cmd += ["-color_primaries", probe.color_primaries] + if probe.color_transfer: + cmd += ["-color_trc", probe.color_transfer] + if probe.color_space: + cmd += ["-colorspace", probe.color_space] + + # Audio: map all audio tracks and preserve bit depth + # -map 0:a maps ALL audio streams from input 0 + cmd += ["-map", "0:a"] + + audio_codec = _audio_codec_for(probe) + cmd += ["-c:a", audio_codec] + + cmd.append(str(output)) + return cmd + def _probe_output_metadata(output: Path) -> tuple[int, int, float]: """Use ffprobe to get exact dimensions and duration of the generated proxy. @@ -151,7 +211,7 @@ def generate_proxy( except OSError as e: raise ProxyError(source, f"cannot create output directory {output.parent}: {e}") from e - cmd = _ffmpeg_cmd(fp, source, output, request.codec, request.target_height) + cmd = _ffmpeg_cmd(fp, source, output, request.codec, request.target_height, probe=request.probe) try: result = subprocess.run(cmd, capture_output=True, text=True, check=False) @@ -255,25 +315,37 @@ def generate_proxies( return ProxyBatchResult(skipped=skipped) ffmpeg_path = find_ffmpeg(cfg) + from media_mate.probe import ProbeError, find_ffprobe, probe_file + + ffprobe_path = find_ffprobe(cfg) command = f"media-mate proxy {source} --out {output_dir}" run_id = store.start_run(command) results: list[ProxyResult] = [] - errors: list[tuple[Path, str]] = [] + failures: list[tuple[Path, str]] = [] + already_existed: list[tuple[Path, Path]] = [] for f, rel in files: out = (output_dir / rel).with_suffix(".mov") try: if out.exists(): - errors.append((f, f"proxy already exists at {out}")) + already_existed.append((f, out)) continue + # Probe the source to get accurate metadata for ffmpeg flags. + probe: MediaProbe | None = None + try: + probe = probe_file(f, ffprobe_path=ffprobe_path) + except ProbeError: + pass # proceed without probe data; _ffmpeg_cmd will use safe defaults + request = ProxyRequest( source_path=str(f), output_path=str(out), codec=cfg.proxy_codec, target_height=cfg.proxy_height, + probe=probe, ) result = generate_proxy(request, ffmpeg_path=ffmpeg_path) results.append(result) @@ -297,25 +369,26 @@ def generate_proxies( ) ) except ProxyError as e: - errors.append((e.path, e.reason)) + failures.append((e.path, e.reason)) except OSError as e: - errors.append((f, str(e))) + failures.append((f, str(e))) - if not errors: + if not failures: status = RunStatus.SUCCESS error_msg = None elif results: status = RunStatus.PARTIAL - error_msg = _format_errors(errors) + error_msg = _format_errors(failures) else: status = RunStatus.FAILED - error_msg = _format_errors(errors) + error_msg = _format_errors(failures) store.finish_run(run_id, status, error_msg) return ProxyBatchResult( results=results, - failures=[ProxyFailure(source_path=str(p), reason=r) for p, r in errors], + failures=[ProxyFailure(source_path=str(p), reason=r) for p, r in failures], skipped=skipped, + already_existed=[ProxySkip(source_path=str(s), proxy_path=str(o)) for s, o in already_existed], ) From 9b1e9fb6cef929c8d98fa775db31e74b0165309d Mon Sep 17 00:00:00 2001 From: Bruce Date: Thu, 9 Jul 2026 23:34:54 -0500 Subject: [PATCH 2/2] v0.3.0: execute all v0.3 spec items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes: - verify: stop mutating baseline on mismatch (#21) - proxy: R3D/BRAW/ARI clear error + fail fast (#24) - proxy: add -fps_mode cfr for VFR→CFR normalization (#8) - resolve: empty source folder guard (#20) - resolve: spec honesty docstring re: limitations (#30) Features: - organize: source-structure-preserving default template (#22) - organize: same-device hardlinking via os.link() (#11) - organize: spanned/multi-file clip detection + warnings (#23) - verify: --accept-changes CLI flag for explicit rebaseline Data model: - OrganizeResult: span_warnings field - OrganizeOpRecord: operation field (copy/move/link) - log.py: operation column added to organize_ops table Spec: - Updated SPEC.md v0.3.0 header + v0.3.0 changelog section - README badge bumped to 0.3.0 --- README.md | 2 +- SPEC.md | 48 ++++++++++++++++- src/media_mate/__init__.py | 2 +- src/media_mate/cli.py | 18 ++++++- src/media_mate/log.py | 6 ++- src/media_mate/models.py | 11 +++- src/media_mate/organize.py | 103 ++++++++++++++++++++++++++++++++----- src/media_mate/proxy.py | 15 ++++++ src/media_mate/resolve.py | 40 ++++++++++---- src/media_mate/verify.py | 30 ++++++++--- 10 files changed, 234 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index fe707c8..10effd0 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # media-mate -[![Version](https://img.shields.io/badge/version-0.2.2-blue)](https://github.com/dspury/media-mate) +[![Version](https://img.shields.io/badge/version-0.3.0-blue)](https://github.com/dspury/media-mate) [![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE) [![CI](https://img.shields.io/github/actions/workflow/status/dspury/media-mate/ci.yml?style=flat-square)](https://github.com/dspury/media-mate/actions/workflows/ci.yml) diff --git a/SPEC.md b/SPEC.md index 784d305..adcf20f 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,8 +1,8 @@ -# media-mate — Spec v0.2.2 +# media-mate — Spec v0.3.0 > **Name:** `media-mate` > **Repo location:** `dspury/media-mate` -> **Version:** 0.2.2 +> **Version:** 0.3.0 > **Status:** Released — stable --- @@ -486,6 +486,35 @@ Closed in v0.2.2: #7 (SAR), #17 (--dry-run), #19 (proxy drops TC/audio), #25 (ed --- +### v0.3.0 — Organizational correctness and spec hardening + +**Status:** Released. + +#### Bug fixes + +- **Verify would silently rebaseline on any mismatch.** When verification found files missing or modified, `store.replace_verification_snapshot()` was unconditionally called, replacing the known-good baseline with the mismatched state. This meant corruption could suppress future detections. Fixed: snapshot is only replaced when verification is clean, or when the user explicitly acknowledges via `verify --accept-changes`. +- **R3D/BRAW/ARI silently produced empty proxies.** These RAW codecs require DecodeClass/clip decoding in Resolve or compositor-level tools — ffmpeg cannot generate correct proxies from them. Fixed: these container formats now fail immediately with a clear error message instead of producing silent quality failures. + +#### New capabilities + +- **Source-structure-preserving default template.** The default organize template changed from `{root}/{codec_family}/{resolution_bucket}/{filename}{ext}` to `{root}/{source_relpath}/{filename}{ext}`. Media is now organized preserving card/scene/take subfolder structure, matching how AEs and DITs think about footage. Users who want codec+resolution grouping can set `template = "{root}/{codec_family}/{resolution_bucket}/{filename}{ext}"` in `media-mate.toml`. +- **Same-device hardlinking in organize.** When `copy` mode is used and source and destination are on the same filesystem (detected via `st_dev`), `os.link()` is used instead of `shutil.copy2()`. Zero-copy on same-device copies; originals remain immutable. Falls back to copy on cross-device or permission errors. +- **Spanned/multi-file clip detection.** Before organizing, the source folder is scanned for multi-file clip patterns (sequential suffixes like `_001.mov`, `_002.mov`, and RED `.RDC` files). Detected groups are logged as warnings (`[SPAN]`) so the user can verify all parts are included. +- **`verify --accept-changes`.** New CLI flag on the verify command. When verification finds differences, the user can re-run with `--accept-changes` to explicitly accept the new state as the new baseline, without having to manually edit the database. + +#### Data model changes + +- `OrganizeResult`: new field `span_warnings: list[str]` — one warning string per detected multi-file clip group. +- `OrganizeOpRecord`: new field `operation: Literal["copy", "move", "link"]` — records whether this op was a copy, move, or hardlink. +- `log.py` `organize_ops` table: new `operation TEXT NOT NULL` column. + +#### Resolve improvements + +- **Empty source guard.** `create_resolve_project()` now raises `ResolveError` immediately if the source folder contains no files, rather than silently creating an empty Resolve project. Message: `"source folder is empty: . Resolve cannot import from an empty folder — add media before trying to create a project."` +- **Spec honesty.** Module docstring updated with explicit known limitations: empty timeline (MediaPoolItem creation deferred), bin naming assumptions, and spanned-clip handling deferred to v1.0. + +--- + ## 19. Open issues — v0.3 candidates The following issues are acknowledged and targeted for v0.3. Each requires a spec change or design decision before implementation. @@ -576,3 +605,18 @@ All items below were approved and shipped in v0.2.2: - v0.3 candidates as documented in §19 Shipped as approved ✓ + +--- + +All items below were approved and shipped in v0.3.0: + +- Verify no-silent-rebaseline (#21) — §18 v0.3.0 +- R3D/BRAW/ARI clear error (#24) — §18 v0.3.0 +- VFR CFR default (#8) — §18 v0.3.0 +- Source-preserving default template (#22) — §18 v0.3.0 +- Same-device hardlinking (#11) — §18 v0.3.0 +- Spanned clip detection (#23) — §18 v0.3.0 +- Resolve empty source guard (#20) — §18 v0.3.0 +- Spec honesty for Resolve (#30) — §18 v0.3.0 + +Shipped as approved ✓ diff --git a/src/media_mate/__init__.py b/src/media_mate/__init__.py index e8b391e..41c0c1f 100644 --- a/src/media_mate/__init__.py +++ b/src/media_mate/__init__.py @@ -1,4 +1,4 @@ """media-mate — Zero-cost CLI for post-production media ops.""" -__version__ = "0.2.2" +__version__ = "0.3.0" __all__ = ["__version__"] diff --git a/src/media_mate/cli.py b/src/media_mate/cli.py index dc01bbb..d8c3f35 100644 --- a/src/media_mate/cli.py +++ b/src/media_mate/cli.py @@ -169,6 +169,10 @@ def organize(ctx: click.Context, path: Path, root: Path, do_move: bool, dry_run: console.print(f" {err}") if len(result.errors) > 5: console.print(f" ... and {len(result.errors) - 5} more") + if result.span_warnings: + console.print("[yellow]Spanned clip warnings:[/yellow]") + for w in result.span_warnings: + console.print(f" {w}") # --------------------------------------------------------------------------- @@ -297,11 +301,17 @@ def resolve_create( @main.command() @click.argument("path", type=click.Path(exists=True, path_type=Path)) +@click.option( + "--accept-changes", + is_flag=True, + default=False, + help="Acknowledge and record the current state as the new baseline after a mismatch.", +) @click.pass_context -def verify(ctx: click.Context, path: Path) -> None: +def verify(ctx: click.Context, path: Path, accept_changes: bool) -> None: """Verify a folder against its previous checksum snapshot.""" store = _get_store(ctx) - report = verify_folder(path, store) + report = verify_folder(path, store, accept_changes=accept_changes) console = Console() if report.is_clean: @@ -314,6 +324,10 @@ def verify(ctx: click.Context, path: Path) -> None: console.print(f" Modified: {report.files_modified}") if report.files_added: console.print(f" Added: {report.files_added}") + console.print( + "\n[dim]Run with --accept-changes to acknowledge these differences " + "and set a new baseline.[/dim]" + ) # Exit with the report's exit code so scripts can switch on it. ctx.exit(report.exit_code) diff --git a/src/media_mate/log.py b/src/media_mate/log.py index d1222ae..e8553de 100644 --- a/src/media_mate/log.py +++ b/src/media_mate/log.py @@ -118,6 +118,7 @@ run_id INTEGER REFERENCES runs(id), source_path TEXT NOT NULL, destination_path TEXT NOT NULL, + operation TEXT NOT NULL, -- copy | move | link codec_family TEXT, resolution_bucket TEXT, file_size INTEGER, @@ -410,12 +411,13 @@ def insert_organize_op(self, record: OrganizeOpRecord) -> int: with self._connect() as conn: cur = conn.execute( "INSERT INTO organize_ops (run_id, source_path, destination_path, " - "codec_family, resolution_bucket, file_size, moved_at) " - "VALUES (?, ?, ?, ?, ?, ?, ?)", + "operation, codec_family, resolution_bucket, file_size, moved_at) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?)", ( record.run_id, record.source_path, record.destination_path, + record.operation, record.codec_family, record.resolution_bucket, record.file_size, diff --git a/src/media_mate/models.py b/src/media_mate/models.py index d37b458..38df99d 100644 --- a/src/media_mate/models.py +++ b/src/media_mate/models.py @@ -81,10 +81,15 @@ class OrganizeConfig(BaseModel): """Top-level organize configuration. Template placeholders: {root}, {codec_family}, {resolution_bucket}, - {filename}, {ext}, {date}. + {filename}, {ext}, {date}, {source_relpath}. + + Default template preserves the source folder structure under dest_root + ({source_relpath}), which matches how AEs and DITs think about media + (cards/scenes/takes). Use {codec_family}/{resolution_bucket} as + an alternative layout when you want codec+resolution grouping. """ - template: str = "{root}/{codec_family}/{resolution_bucket}/{filename}{ext}" + template: str = "{root}/{source_relpath}/{filename}{ext}" on_conflict: Literal["skip", "overwrite", "rename"] = "skip" mode: Literal["copy", "move"] = "copy" @@ -100,6 +105,7 @@ class OrganizeResult(BaseModel): duration_seconds: float dry_run: bool errors: list[str] = Field(default_factory=list) + span_warnings: list[str] = Field(default_factory=list) # multi-file clip detections class OrganizeOpRecord(BaseModel): @@ -109,6 +115,7 @@ class OrganizeOpRecord(BaseModel): run_id: int source_path: str destination_path: str + operation: Literal["copy", "move", "link"] # link = hardlink (same-device) codec_family: str | None resolution_bucket: str | None file_size: int | None diff --git a/src/media_mate/organize.py b/src/media_mate/organize.py index ea25154..9ce17ce 100644 --- a/src/media_mate/organize.py +++ b/src/media_mate/organize.py @@ -17,9 +17,12 @@ from __future__ import annotations +import os +import re import shutil from datetime import UTC, datetime from pathlib import Path +from typing import Literal from media_mate.log import LogStore from media_mate.models import ( @@ -78,6 +81,34 @@ class OrganizeError(Exception): "pcm_f32le": "audio", } +# Patterns that suggest a multi-file / spanned clip: +# e.g. ClipName_001.mov, ClipName_002.mxf, ClipName.RDC +_SPANNED_PATTERNS = ( + r"^(?P.+?)[\._](?P\d{3,6})\.[a-zA-Z0-9]+$", + r"^(?P.+?)\.RDC$", +) + + +def _spanned_clip_groups( + files: list[Path], +) -> list[tuple[str, list[Path]]]: + """Detect groups of multi-file / spanned clips and return (base_name, parts). + + Returns a list of (base_name, [paths...]) where each group has 2+ parts. + Checks for sequential numeric suffixes (_001, _002, ...) and RED .RDC files. + """ + groups: dict[str, list[Path]] = {} + for f in files: + name = f.name + for pat in _SPANNED_PATTERNS: + m = re.match(pat, name, re.IGNORECASE) + if m: + base = m.group("base") + groups.setdefault(base, []).append(f) + break + + return [(base, parts) for base, parts in groups.items() if len(parts) >= 2] + def codec_family(codec: str | None) -> str: """Map a codec name to a coarse family bucket. @@ -121,12 +152,27 @@ def build_destination_path( family: str, bucket: str, date: str | None = None, + source_root: Path | None = None, ) -> Path: """Render an organize template to a destination Path. Template placeholders: {root}, {codec_family}, {resolution_bucket}, - {filename}, {ext}, {date}. + {filename}, {ext}, {date}, {source_relpath}. + + source_relpath is the directory part of the source file relative to + source_root (i.e., source's parent directory relative to source_root), + preserving any subfolder structure under the organize root. """ + source_relpath = "" + if source_root is not None: + try: + # source is the file path; source_relpath is its directory + # relative to source_root (preserves card/scene subfolders) + source_relpath = str(source.parent.relative_to(source_root)) + except ValueError: + # source is not under source_root + source_relpath = "" + ctx = { "root": str(dest_root), "codec_family": family, @@ -134,6 +180,7 @@ def build_destination_path( "filename": source.stem, "ext": source.suffix, "date": date or datetime.now(UTC).strftime("%Y-%m-%d"), + "source_relpath": source_relpath, } return Path(template.format(**ctx)) @@ -196,6 +243,16 @@ def organize_path( files = sorted(p for p in source.rglob("*") if p.is_file()) + # Detect multi-file / spanned clips before organizing (logged as warnings) + span_warnings: list[str] = [] + spanned = _spanned_clip_groups(files) + for base, parts in spanned: + span_warnings.append( + f"[SPAN] {base}: {len(parts)} files detected as multi-file clip " + f"({', '.join(p.name for p in parts)}); " + f"organizing individually — verify all parts are included" + ) + if not files: return OrganizeResult( source_path=str(source), @@ -233,7 +290,9 @@ def organize_path( family = codec_family(probe.codec) bucket = resolution_bucket(probe.height) - dest = build_destination_path(cfg.template, dest_root, f, family, bucket) + dest = build_destination_path( + cfg.template, dest_root, f, family, bucket, source_root=source + ) # Conflict handling if dest.exists(): @@ -250,18 +309,34 @@ def organize_path( if do_move: shutil.move(str(f), str(dest)) else: - shutil.copy2(str(f), str(dest)) - store.insert_organize_op( - OrganizeOpRecord( - run_id=run_id, - source_path=str(f), - destination_path=str(dest), - codec_family=family, - resolution_bucket=bucket, - file_size=size, - moved_at=datetime.now(UTC), + # Same-device: use os.link() for zero-copy. + # Both paths must be on the same filesystem (stat st_dev). + # Hardlinks share the same inode; originals remain immutable. + same_device = f.stat().st_dev == dest.parent.stat().st_dev + if same_device: + try: + os.link(str(f), str(dest)) + op: Literal["copy", "move", "link"] = "link" + except OSError: + # Fallback to copy if hardlink fails (cross-fs, permissions) + shutil.copy2(str(f), str(dest)) + op = "copy" + else: + shutil.copy2(str(f), str(dest)) + op = "copy" + + store.insert_organize_op( + OrganizeOpRecord( + run_id=run_id, + source_path=str(f), + destination_path=str(dest), + operation=op, + codec_family=family, + resolution_bucket=bucket, + file_size=size, + moved_at=datetime.now(UTC), + ) ) - ) files_moved += 1 bytes_moved += size @@ -296,6 +371,7 @@ def organize_path( duration_seconds=duration, dry_run=dry_run, errors=errors, + span_warnings=span_warnings, ) @@ -305,4 +381,5 @@ def organize_path( "codec_family", "organize_path", "resolution_bucket", + "_spanned_clip_groups", ] diff --git a/src/media_mate/proxy.py b/src/media_mate/proxy.py index 74fe30a..678f1ba 100644 --- a/src/media_mate/proxy.py +++ b/src/media_mate/proxy.py @@ -168,6 +168,12 @@ def _ffmpeg_cmd( audio_codec = _audio_codec_for(probe) cmd += ["-c:a", audio_codec] + # Force CFR on VFR sources (action cams, phone recordings, screen captures). + # r_frame_rate from ffprobe is the real rate; avg_frame_rate is nominal. + # VFR sources cause audio sync drift in proxies — CFR normalization fixes it. + # -fps_mode cfr is applied after input decode, before output encode. + cmd += ["-fps_mode", "cfr"] + cmd.append(str(output)) return cmd @@ -347,6 +353,15 @@ def generate_proxies( target_height=cfg.proxy_height, probe=probe, ) + + # Reject RAW codecs that stock ffmpeg cannot decode. + # These require vendor SDKs (RED, Blackmagic, ARRI). + # Container is recognized but decode will fail with a cryptic error. + raw_codecs = {"r3d", "braw", "ari"} + if probe and probe.video_codec and probe.video_codec.lower() in raw_codecs: + failures.append((f, f"RAW codec '{probe.video_codec}' requires vendor SDK; stock ffmpeg cannot decode")) + continue + result = generate_proxy(request, ffmpeg_path=ffmpeg_path) results.append(result) diff --git a/src/media_mate/resolve.py b/src/media_mate/resolve.py index 5e7e927..fcf0c9c 100644 --- a/src/media_mate/resolve.py +++ b/src/media_mate/resolve.py @@ -1,8 +1,8 @@ """Resolve capability — create DaVinci Resolve projects programmatically. -Two layers, both always available: +Architecture: manifest-first, Resolve API best-effort. -1. **Manifest builder** (pure functions, no Resolve dependency): +1. **Manifest builder** (always runs, pure functions, no Resolve dependency): resolve_bin_structure(source_folder) -> list[str] Compute bin paths mirroring the folder's subdirectory layout. build_project_manifest(source_folder, spec, proxy_dir=None) -> dict @@ -10,17 +10,27 @@ write_manifest(manifest, output_path) -> Path Write a manifest dict to disk. -2. **Resolve adapter** (best-effort, uses the live API when available): +2. **Resolve adapter** (best-effort — requires live Resolve + running API): find_resolve(config) -> ModuleType | None Try to load the DaVinciResolveScript module; None if unavailable. create_resolve_project(spec, source_folder, proxy_dir, store, config=None) -> ResolveProjectResult - Try the Resolve API; fall back to writing a manifest file when Resolve - isn't available. Always returns a result; ``resolve_version=None`` - indicates the manifest fallback was used. - -The manifest is also what gets written on disk when the Resolve API is -unavailable, so a user can manually create the project in Resolve from the -manifest description. + Always builds the manifest first. Then tries the Resolve API; falls back + to writing a manifest file when Resolve isn't available or an API call fails. + ``resolve_version=None`` indicates the manifest fallback was used. + +The manifest is the ground truth — it is always written to disk (at +``.manifest.json``) when the Resolve API path fails or is +unavailable, so the user can manually act on it. + +**Known limitations (deferred to v1.0):** +- MediaPoolItem creation: Resolve requires MediaPoolItems (not raw file paths) + to link clips into the timeline. The current API path creates an empty timeline; + clip-by-clip linking requires a heavier import step that is deferred. +- Bin naming: sub-bins are named as ``"root/subfolder"`` (e.g. ``"raw/shoot_day_1"``). + This may not match the user's existing bin structure in Resolve. +- Spanned/multi-file clips: each file is treated as a separate clip. A proper + spanned-clip model requires understanding clip relationships per camera format + (RED R3D multi-part, ARRI RAW .ari + .idx, etc.) — deferred to v1.0. """ from __future__ import annotations @@ -300,6 +310,16 @@ def create_resolve_project( if not source_folder.is_dir(): raise ResolveError(f"source folder does not exist: {source_folder}") + # Guard: an empty source folder produces an empty Resolve project, which + # is never what the user wants. Reject early with a clear message rather + # than silently creating a blank project they have to delete. + media_files = _media_files_in(source_folder) + if not media_files: + raise ResolveError( + f"source folder is empty: {source_folder}. " + "Resolve cannot import from an empty folder — add media before trying to create a project." + ) + manifest = build_project_manifest(source_folder, spec, proxy_dir_resolved) command = f"media-mate resolve create {source_folder} --project {spec.name}" diff --git a/src/media_mate/verify.py b/src/media_mate/verify.py index a8fafff..5fad230 100644 --- a/src/media_mate/verify.py +++ b/src/media_mate/verify.py @@ -4,14 +4,15 @@ compute_checksum(path, algo="xxhash") -> str Compute a checksum for a single file. Streams in 64KB chunks so it's safe for large media files. - verify_folder(folder, store, config=None) -> VerificationReport + verify_folder(folder, store, config=None, accept_changes=False) -> VerificationReport Compute current checksums for a folder, diff against the previously - recorded snapshot, write the new snapshot, return a structured report. + recorded snapshot. Returns a structured report. Workflow: First call: snapshot is created (no prior baseline), report shows 0 diffs. - Subsequent calls: each call's snapshot becomes the new baseline; the report - shows what changed since the previous call. Designed for cron. + Subsequent calls: diff against the previous snapshot. The baseline is + NOT updated on mismatch — use --accept-changes to explicitly set a new + baseline after reviewing the diff. Designed for cron. Exit codes (per SPEC.md §5.5; priority-ordered): 0 = clean (no diffs, or first-time snapshot) @@ -121,12 +122,18 @@ def verify_folder( folder: Path, store: LogStore, config: MediaMateConfig | None = None, + *, + accept_changes: bool = False, ) -> VerificationReport: """Verify a folder against the previous snapshot; write a new snapshot. First call for a folder: creates a snapshot, returns a clean report. - Subsequent calls: diff against the previous snapshot, write a new - snapshot, return a report describing what changed. + Subsequent calls: diff against the previous snapshot. + + The baseline is NOT updated on mismatch (immutable baseline). This prevents + a corrupted file from silently overwriting the known-good baseline. + Use --accept-changes (accept_changes=True) to explicitly acknowledge + a new baseline after reviewing the diff. The verification itself is logged to the runs + verifications tables in the audit log, so the run history is queryable. @@ -175,8 +182,15 @@ def verify_folder( common = prev_paths & new_paths modified = sorted(p for p in common if prev[p] != new[p]) - # Persist the new snapshot (replaces old) - store.replace_verification_snapshot(folder_str, new_rows) + # Persist the new snapshot only when verification is clean, or when + # the user explicitly acknowledges the new baseline via --accept-changes. + # On mismatch without acceptance, the baseline is immutable — corruption + # does not suppress future detections by overwriting the known-good snapshot. + exit_code = _exit_code( + missing=bool(missing), modified=bool(modified), added=bool(added) + ) + if exit_code == 0 or accept_changes: + store.replace_verification_snapshot(folder_str, new_rows) # Log the run command = f"media-mate verify {folder}"