feat(editor): audio-kind assets + two-column panel UX (timeline video editor v1) - #57
Merged
Conversation
Implements the first two milestones of the timeline video editor PRD
(video-editor-prd.md):
M0 — Scaffolding
- Thread.type 'editor' + EditorDocument data model in shared/types.ts
(MediaAsset, Clip, Track, TimelineItem incl. speed/retime, persona &
TimelineDiff stubs)
- ThreadManager.createEditorThread (no chat preprocessing auto-start) and
updateThreadWith queued mutator for race-free concurrent doc writes
- /editor/:id route, Home "Video Editor" card, type-based thread routing
- Full-bleed 4-zone editor shell (media / preview / inspector / timeline)
with editorStore (ownership-split autosave vs thread-updated echoes)
M1 — Media import + per-asset preprocessing + selectable pieces
- src/main/editor: per-asset orchestrator (proxy -> scenes -> clips ->
progressive thumbnails), K=3 concurrency cap, per-asset abort, task ids
namespaced `${assetId}:step`, opt-in Gemini scene descriptions reusing
extraction.generateSceneDescription via an asset-scoped context
- IPC: create-editor-project, save-editor-doc, add-media-asset,
import-media-url (per-asset progress), remove-media-asset,
preprocess-media
- Media panel with live per-step progress, error isolation + retry,
interrupted-run resume; clip tray with selectable scene pieces;
preview monitor with clip in/out playback; inspector
- Editor-aware repairThreadPaths; scenedetect threshold param
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements PRD milestone M2 — the editor now edits:
Timeline & manual tools
- Purpose-built multi-track timeline (pxPerSecond model, shared scroll
container for ruler+lanes, adaptive tick steps, windowed clip mounting)
- Drag pieces from the tray onto tracks (HTML5 DnD) + keyboard placement
- Pointer-gesture state machine (useTimelineInteractions): move with
snapping (item edges/playhead/markers) and cross-track validation,
edge-trim with neighbor clamping (Alt = ripple-trim), ruler scrubbing,
Escape cancels; one history step per gesture
- Split at playhead, delete vs ripple-delete (magnetic default), nudge,
markers, zoom anchored at cursor (4-400 px/s) + fit-to-window/selection,
track mute/lock/hide + add-overlay-track
- Retime/speed: constant per-clip speed 0.25-4x with numeric speed AND
target-duration fields in the inspector; on-timeline duration =
(out-in)/speed; downstream ripple; hatched visual + speed badge
- Keyboard map (space, arrows, S/Cmd+B, Delete, M, Cmd+Z/Cmd+Shift+Z,
+/-) guarded against typing contexts; ARIA roles + live seek announcements
EDL preview
- useEdlPlayback: A/B double-buffered <video> pair plays the composed
timeline — boundary switching with preload, black gap advance on wall
clock, playbackRate honors retime, item/track mute respected
- PreviewMonitor: Source | Timeline modes with auto-switching
- Filmstrip <-> Context view toggle on timeline clips (scene thumbnail
vs visual description)
Undo/redo
- src/shared/timeline.ts: pure snapshot diff engine (forward/inverse
TimelineDiff) + validated apply (schemaVersion, id/range checks,
speed clamp, duration recompute) — reused by M3 AI diffs later
- src/main/editor/history.ts: sidecar userData/editor-history/{id}.json
with 50-step ring, sparse keyframes, redo-branch truncation, atomic
writes; undo pointer persists inside the doc autosave patch so doc
and pointer stay crash-consistent; history survives restart
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements PRD milestone M3 — the AI co-editor:
Personas
- 9 built-in personas in src/main/constants/personas.ts, grouped
long-form (Podcast Editor default, Long-Form Polisher, Silence
Cleaner, Chapter Organizer, Study-Notes; targetDurationSec null =
length-preserving) and summarize (Concise, Highlight Reel,
Storyteller, Social Shorts). Built-ins are code, merged at read time,
never persisted; user personas live globally in settings.json
(get/set-personas IPC) with builtin-collision filtering
- PersonaPicker popover (grouped, clone/edit/delete) + PersonaEditorModal
(create/clone/edit, preserve-runtime checkbox, live summary line)
Prompt engine
- Constrained EditorOps schema: the model returns removeItemIds /
updateItems (whitelisted fields) / addClips (assetId + scene # refs) /
addMarkers / rationale / optional answer — never raw TimelineDiff;
opsToDiff() maps server-side (generated item ids, derived durations,
unknown refs pruned into droppedOps)
- Full context windowing (src/shared/ai-scope.ts + src/main/editor/
context.ts): selection scope -> chapter/marker window (engages past
40min/400 items, 150-item cap) -> full; per-asset scene caps with
60->30->12 degradation ladder; explicit truncation + thin-context
flags, never silent
- composeSystemInstruction: persona voice + fixed editor contract +
defaults (null duration -> PRESERVE full runtime branch)
- One structured Gemini call per turn ('editor-edit' model selection),
streamed via editor-turn-update onto persisted PromptTurn records;
abortable; usage/cost recorded to thread.usageHistory
Review UX
- Proposals render ghosted on the timeline: adds = dashed sparkle
clips, removals = struck-through, moves = ghost twins at the new
position; proposed chapter markers dashed on the ruler
- ProposalCard: rationale, op-count chips, dropped-ops/thin-context/
truncation notices, usage+cost, Accept-all / Reject
- Double validation (at receipt and at accept) so manual edits during
review never mis-apply; accept commits ONE ai-origin history step
(turnId + resultStepId stamped) — a single undo reverts the AI edit
- PromptBar: persona chip, auto-grow input, scope chip with
Selection/Chapter/Whole override, send/stop, inline error + retry;
question turns show a transient answer card
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements PRD milestone M4 — the editor now exports real video files, completing the v1 loop (import -> split -> edit -> export): Render engine (src/main/editor/render.ts) - Fast path: single-source, speed-1, unmuted, gapless timelines map to TimelineSegment[] (toFixed(3) — String() exponent forms would break timeToSeconds) and reuse assembleVideo unchanged - Region path (PRD option C, segment-then-concat): the sequence slices into clip/gap regions; each renders a UNIFORM mp4 intermediate (h264 videotoolbox 8M on mac / libx264 crf18 elsewhere + aac 48kHz stereo, -video_track_timescale 90000, both A+V streams always present) normalized to source-derived WxH/FPS (aspect-preserving pad — NOT the blindly-seeded timelineMeta); stitched via concat demuxer -c copy +faststart with a one-shot re-encode fallback - Retime: setpts=(PTS-STARTPTS)/speed + chained atempo (0.5-2.0 per stage; asetrate when preservePitch=false); apad + output -t pin A/V to exact region duration - Gaps render black+silence (color/anullsrc lavfi); muted items/tracks and audio-less sources get anullsrc so concat stream layout never varies; input-side -ss/-t seeking for speed with frame accuracy - Pre-flight: regions computed + all source files existence-checked synchronously — bad exports reject the invoke, no ghost renders - Duration-weighted progress (regions 0-96, stitch 96-100) over new editor-render-progress event; abort kills ffmpeg + cleans workdir; renders auto-abort on project delete Overlap hardening (bug found in verification) - AI-accept placements weren't gesture-clamped and could overlap items: new repairOverlaps() (shared/timeline.ts) pushes later items right until sequential; runs on proposal apply AND on project load (heals existing docs); export self-repairs by butting instead of throwing; left-edge ripple-trim now also clamps against its left neighbor Export UX - ExportDialog: quality choice (original / 480p proxy preview) -> live progress + cancel -> done (Save As... via save-video, Open project folder) / neutral canceled state; closing the dialog does NOT abort — render state lives in the store - Header Export button with live "Exporting N%" label; disabled only when the timeline is empty; overlay/text-track notice line Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…al flow
Replaces the M3 ghost-preview + Accept/Reject flow with a branchable
revision tree, mirroring the chat editor's every-generation-is-a-version
paradigm:
Prompt flow rework
- An AI prompt result now APPLIES IMMEDIATELY (validated + overlap-
repaired) and lands as a new revision branching from the current one;
the timeline switches to it. AiResultCard shows "Applied - V{n}" with
rationale/counts/cost/notices and a "Back to V{parent}" jump.
Rejecting = switching back; the result stays in the tree.
- Unsaved manual work is silently auto-checkpointed BEFORE the AI
revision is created, so the AI revision's parent snapshot is exactly
the state the diff applied to and hand edits are never stranded.
- Cmd+Z still undoes an AI diff in place (ring step preserved); the
fine-grained ring resets only on revision SWITCHES (stale diffs would
corrupt a switched snapshot via redo) — new clear-editor-history IPC.
- Removed: pendingProposal state, ghost computeds/props/styling in
TimelineClip/TrackLane/TimelineRuler, ProposalCard, accept/reject.
Revision tree core
- EditorRevision {id, parentId, seq, origin init|ai|manual, label,
turnId/personaId, FULL snapshot incl. markers} in a new sidecar
userData/editor-revisions/{threadId}.json (temp-then-rename writes,
own monotonic revisionCounter so V numbers stay clean and never
reused, cap 100 with oldest-LEAF-only pruning, root protected).
Snapshots are self-contained by design — never diff-chained into the
capped undo ring. Doc carries only currentRevisionId.
- Lazy root bootstrap ("Original") on first checkpoint/prompt — zero
migration; sidecar loss never mutates the working doc.
- switchRevision: dirty guard (Save & switch / Discard / Cancel),
snapshot restore incl. markers, ring reset, selection prune, playhead
clamp. Subtree delete via recursive parent-pointer collect (root
undeletable; current relocated to the subtree's parent first).
Switch/delete blocked while a prompt runs (parentage correctness).
UI
- Right rail now tabbed Inspect | Revisions (badge dot when an AI
revision lands). RevisionsPanel: DFS tree list with depth indent,
V{n} pills, origin icons (persona emoji / bookmark / flag), first-
clip thumbnails, relative times, current ring + dirty asterisk,
hover subtree-delete. SaveRevisionButton with inline label popover
("No changes since V{n}" when clean) + timeline-toolbar bookmark.
- RevisionGraphModal: standalone Vue Flow embed (isolated instance) —
tidy tree layout (main line down, branches fan right, leaf-column
assignment), revision cards with 3-thumb strips, click-to-switch
with the modal staying open for branch-hopping, fitView on open.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Register @playwright/mcp (.mcp.json) attaching over CDP at :9222 so the app's Vue UI can be driven and screenshotted for validating changes. Add `start:debug` script to build and launch Electron with the remote debugging port open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… editor PRD Adds the §0 status ledger (M0–M4 + revisions shipped, per-milestone commits, not-yet-built list), rewrites §5.7 to the apply-immediately revision flow (§5.7a), and updates milestones/traceability to match what shipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pletion) Phase 1 — transcript → clip.text (PRD §5.2): - New opt-in 'audio' + 'transcript' preprocess steps reusing the chat extraction phases verbatim through the asset-scoped PipelineContext - Transcript excerpts merged into each Clip.text by [in,out] time overlap, preserved across scene re-runs; enriches AI context beyond visual-only - Inspector asset mode gains a "Transcribe (uses Gemini)" opt-in button; the existing clip-mode Transcript block now populates Phase 2 — silence finder (PRD §5.6, assistive/review-only): - detectSilence() ffmpeg helper (silencedetect stderr parsing) + shared SilenceRegion type + find-silence IPC channel - SilenceFinder.vue: tunable noise-floor/min-duration thresholds, scan, reviewable region list with seek-on-click, explicit ripple-delete apply - Scan state lives in editorStore keyed by asset, so results stay visible across Inspector modes (asset/clip/item) while previewing pieces - applySilenceRegions carves regions out of placed items per track, clamped to item bounds, closes gaps, single undoable history step Verified end-to-end in the app against a synthetic speech/silence video: detected regions match ffmpeg ground truth, ripple math exact (27.44s → 17.37s), undo restores across restart, transcription windowed correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ions PRD §5.2 corrective controls (editor v1 completion, phase 3): - ClipTray header gains merge-selected / split-selected buttons and a collapsible detector panel (threshold slider 5–60 + Re-detect) that re-runs detection via the already-wired preprocess-media threshold param - mergeClips/splitClip main-process mutations (clips are main-owned): adjacency-validated merge with joined visual/text, midpoint split with min-piece guard; both reindex and clear masterSegmentIndex - Validation runs before the queued mutator so rejections reach the renderer (updateThreadWith swallows mutator throws); the mutator re-checks and skips on mid-flight changes - Inline tray error surfaces rejected operations Verified live: merge [0,5]+[5,10]→[0,10] with joined transcript, split back to halves, non-adjacent merge rejected with visible message, re-detect rebuilds pieces and preserves epsilon-matched transcripts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…vigation) PRD §5.3 (editor v1 completion, phase 4): - TimelineMinimap: always-visible overview strip under the toolbar mapping the full sequence — per-track item blocks, marker ticks, playhead line, and a visible-window rectangle; click/drag scrolls the main timeline (navigation only, never moves the playhead) - Toolbar gains a Markers & chapters popover: sorted marker list with label + timecode, jump-on-click (seek + scroll into view), per-row remove, and a press-M empty-state hint Verified live: minimap window rect tracks zoom/scroll, 90% strip click jumps scrollLeft proportionally, marker jump lands the playhead at the marker time, markers persist across autosave. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRD §5.11 (editor v1 completion, phase 6): a global media query disables fade/pulse/shimmer animations, hover lifts, press scales, and smooth scrolling when the OS requests reduced motion; spinners slow instead of stopping (they convey busy-state), and functional motion (playback, the playhead, progress widths) is untouched. scroll-behavior needs !important because a bundled third-party sheet re-declares it after our block. Verified via emulated reducedMotion in both directions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRD §5.5 (editor v1 completion, phase 5): - generateFilmstrip(): ONE ffmpeg pass (fps=1/interval, 120px) replaces the process-per-frame extractFrame for strips; interval = max(1s, dur/300) so a multi-hour source stays a single bounded run - New 'filmstrip' preprocess step (reuses the preprocess-media channel and the K=3 heavy-slot cap) populating the previously dormant MediaAsset.filmstrip; cached per asset under frames/strip/ - editorStore lazily requests strips when filmstrip view is active for completed assets placed on the timeline - TimelineClip tiles one lazy-loaded frame per ~64px slot across the clip's source range (zoom-adaptive density, ≤60/clip), falling back to the scene thumbnail until the strip lands Verified live: 27 frames auto-generated on view, clips tile floor(w/64) frames at every zoom (87px→1, 301px→4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six of seven deferred items shipped and verified (transcript, silence finder, sensitivity + merge/split, minimap + jump list, dense filmstrip, reduced-motion); audio-kind assets remain, scoped as its own milestone with the render-engine region-slicing prerequisite spelled out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TimelineClip's hovering flag was set on pointerdown but never cleared — no pointerenter/pointerleave existed — so once a clip had ever been clicked its edge trim handles rendered permanently, visually swallowing narrow clips. Wire real hover tracking and drop the pointerdown hack. Verified live: handles absent on untouched clips, present while selected/hovered, gone after deselect once the pointer leaves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Popover and playhead were both z-30, and the playhead sits later in the DOM, so it painted through the list. Raise the popover to z-50 (backdrop z-40), above every in-panel layer (clips z-10/20, snap guide z-20, playhead z-30). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every long-running step now reports percent or count instead of an
indeterminate spinner:
- scenes: parse PySceneDetect's tqdm 'Progress: N%' from streamed stderr
(emitted even without a tty) into the task bar
- audio + descriptions: derive percent from the reused phases' status
strings ('Converting… 45%', 'Analyzing scenes 51 to 100 / 418')
- transcript: honest stage milestones (10 upload/transcribe → 85 merge →
100) since a single Gemini call has no incremental signal
- filmstrip + silence scan: ffmpeg progress events; silence streams
'editor-silence-progress' over IPC and the scan button shows 'Scanning… N%'
- progress writes throttled to ≥5% (task) / ≥2% (event) steps since every
task update persists the thread JSON and broadcasts
Verified live on a 3-minute import: scenes streamed 0→100 in ~10 steps,
audio 0→48→100, transcript 10→85→100, silence events delivered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five UX fixes in one coherent pass:
1. Transcript runs UP FRONT: DEFAULT_STEPS now proxy → audio → transcript
→ scenes → thumbnails, with audio/transcript as soft-fail steps (a
missing Gemini key or quota error falls back to scene pieces instead of
bricking the import).
2. Pieces are REAL editorial segments: clips derive from the transcript
(one piece per spoken statement + [Silence] pieces over gaps, mirroring
timeline/enrichment.ts), so the tray shows what was said, pre-populated
with text. Scene detection stays as the fallback and as the explicit
sensitivity-re-run path; thumbnails now follow clips, not scenes.json.
3. Collapsible rails: Media (left) and Chat/Inspect/Revisions (right)
collapse to slim vertical strips, persisted in localStorage; the
monitor takes the reclaimed width.
4. Header overlap fixed: Export teleports into #header-actions-portal so
it composes with the settings/theme cluster instead of colliding.
5. Prompt box → chat panel: new ChatPanel renders the full persisted turn
history (user bubbles with scope+time, answers, applied-edit summaries
with V{n}/op-count/cost chips, error+retry, running indicator) with the
restacked PromptBar pinned at the bottom of the right rail; chat is the
default tab with an unseen-activity badge.
Verified live: 3-min speech import produced 55 text-bearing pieces
(36 speech + 19 silence) with per-piece thumbnails; rails collapse/expand
(monitor 552→796px); a question turn round-tripped through the chat panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egion The chat input stacked card-inside-card (panel border → PromptBar card → boxed persona/scope chips); flatten it to the modern chat-input shape: - PromptBar loses its own card; the panel's border-t is the only frame. Textarea on top, one quiet meta row below (persona · scope as tiny text-level controls, small send circle right) - PersonaPicker trigger: boxed chip → icon+name text control - Right-rail tabs: bordered pill container → naked segmented text tabs - AiResultCard: floating bordered card → flat tinted message block - AssetRow: border removed; hover tint, ring only when selected Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The scope control now owns the row's flexible space (flex-1 min-w-0) so its label truncates with an ellipsis before reaching the send button, which keeps a fixed ml-2 gap instead of ml-auto. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Editor dropdowns used animate-fade-in-up (fadeInUp 0.5s, translateY 10px), which reads as a slow bottom-to-up slide. Add a dedicated animate-menu (menuIn 0.12s: fade + faint 0.97→1 scale) and apply it to the persona picker, scope menu, save-revision menu, and marker jump list — each with origin-top/origin-bottom so it grows from its anchor edge. Reduced-motion neutralizes it alongside the other decorative animations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AI needs pieces/transcript to reason about the footage, so the prompt input was inert-but-clickable before any media was ready. Add store getters hasReadyMedia (>=1 asset completed with clips) and mediaProcessing, disable the textarea + send until hasReadyMedia, and give both the input placeholder and the chat empty-state three honest states: import media → preparing (spinner) → ready. Verified live: input disabled with no media and while preprocessing, enabled once pieces exist, and a prompt then round-trips to a turn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-row textarea clipped the long guidance placeholders in the narrow right rail. Shorten all four to one-line hints (the chat empty-state already carries the full guidance). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Picks up the Modal fixes (empty controlled-trigger button removed, dark-theme close button no longer a light box) from the published release. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat: timeline video editor (v1 + completion + UX pass)
…ename Editor projects were stuck at the hardcoded 'Untitled Project' with no way to change it. Hybrid fix: - Auto-name: createMediaAsset sets the project title from the first imported asset's filename (extension stripped, underscores → spaces) while the title is still the default — atomic in the same queued mutator that pushes the media, mirroring how chat threads name from the video. - Manual rename: GraphHeader gains an opt-in `editable` prop; the editor passes it so the title becomes click-to-edit (pencil on hover, Enter commits, Esc/blur cancels) → editorStore.renameProject → new rename-editor-project IPC → threadManager.updateThread. The shared graph header is unchanged when editable is not set. Title is a main-owned Thread field, so it persists and echoes via thread-updated. Verified live: import auto-names 'progress_test.mp4' → 'progress test'; manual rename commits on Enter, cancels on Esc, survives reload. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 480p proxy re-encoded every source frame with software decode, which dominates on large/high-fps footage. Two source-fps-aware wins: - Cap proxy fps at 30 when the source exceeds it (a proxy for scrubbing/ thumbnails/scene-detection never needs more) — roughly halves encode time on 60fps footage (~-49%). Guarded so a <=30fps source is never upsampled, which would ADD frames. - Hardware-accelerated decode (videotoolbox) on Mac for the <=30fps path (~-25%), with a one-shot software-decode fallback so a hw-decode-hostile input still produces a proxy instead of erroring. Not combined with the fps cap — benchmarks show the per-frame GPU→CPU download makes it slower once frames are already being dropped. Audio and output size are unchanged; toLowResolution's new opts arg is optional so the chat pipeline path is unaffected (and gains hw decode). Benchmarked on a 2-min 1080p60 source: 37.5s → 19s encode; verified in-app end to end (60fps → 480p/30fps; 25fps+audio → 480p/25fps, audio preserved). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Importing a local file ran fs.copyFileSync of the whole source into tempDir/media/<id>/source/. For a 10–15 GB file that synchronous copy blocks Electron's main process for minutes, freezing the UI and the file dialog — even though the editor only needs the path. Local imports now reference the user's file where it lives (no copy): originalPath = the picked path. Everything downstream already works with an out-of-tempDir path — proxy/thumbnails/export read originalPath directly, media:// serves any absolute path, removeAsset only clears the asset dir (never the user's file), and path-repair leaves non-tempDir paths alone. URL/YouTube imports are unchanged (they download straight into the asset dir, so the copy branch is a no-op for them). Verified: add-media-asset returns in ~200ms instead of blocking on the copy; originalPath references the source; preprocessing still builds the proxy in tempDir. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a per-asset way to delete derived Gemini data. clearAssetData (main) deletes the backing files (the whole per-asset transcripts dir for transcript; scene_descriptions.json for descriptions — scenes.json kept), clears the preprocessing path fields and the derived clip field (text / visual) on every piece, and drops the step's namespaced task. The pieces (segmentation) are preserved; audio is kept so a later re-transcribe skips re-extraction. Wired through a clear-asset-data IPC + editorStore.clearAssetData (with a token-cost confirmation). The Inspector's 'Transcript ready' / 'Scene descriptions ready' states each gain a Remove link; removing reverts to the Transcribe / Describe button so it can be regenerated. Verified live: 56 text-bearing pieces → 0 on remove, files deleted from disk, task dropped, pieces intact, and the Inspector toggles ready ↔ re-run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… (stage a) Lets audio files become first-class MediaAssets on the A1 lane: - probe-based kind detection (video/audio) on local + URL import; audio-safe ffprobe (getAudioMetadata) that tolerates no video stream - kind-aware preprocessing: audio runs audio+transcript only, with a whole-file clip fallback when no transcript (no Gemini key) - audio icons/copy across MediaPanel/AssetRow/ClipTray/ClipTile/TimelineClip; scene-detector controls hidden for audio - preview: audio-track items play through a slaved <audio> element with a blank "audio only" monitor state (useEdlPlayback + audioSegments) - per-item gain slider in the inspector (undoable via editorStore.setItemGain) Export mixing of audio-track items lands in stage b. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ge b) Refactors the segment-then-concat renderer to handle audio-track items: - computeRegions now slices at the UNION of video AND audio item boundaries and sub-slices each covering item onto the region (retime-aware mapping) - unified Region model: optional video slice (or black) + N audio sources - renderRegion builds a k>1 amix graph (normalize=0, per-item gain), with every stream apad+atrim'd to exact region length so A/V stays concat-aligned; 0 sources → anullsrc silence, 1 → the stream, k>1 → amix - audio can extend past video into black regions; audio-only timelines export - fast path (assembleVideo) now skipped whenever any audio-track item exists Filtergraph shapes validated end-to-end against ffmpeg (2-stream amix, black+silence, retimed audio → concat → 1 video + 1 stereo audio). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lback) Audio no longer lands as a single whole-file clip. A new local 'segment' preprocess step (AUDIO_STEPS=['segment']) runs silencedetect and divides the file into CONTIGUOUS pieces at silence midpoints, subdividing long runs into ~20s chunks — so music (no silence) still splits into even chunks and speech splits at pauses. No Gemini needed; transcription stays opt-in. segmentAudio() is pure and unit-checked (music/speech/all-silence/tiny → full coverage). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- AssetRow is now draggable → drops the WHOLE asset as one full-span item (itemFromAsset + store.addItemFromAsset), kind-matched to its lane - TrackLane accepted only video lanes before, so audio pieces could never be dropped on A1 at all; it now accepts video AND audio lanes and validates the dragged kind via a readable kind-hint DnD type (clip or asset payload) - addItemFromClip defaults the target lane by kind, so Enter-to-place and drag both route audio pieces to the audio lane instead of the video lane Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t pool A single shared <audio> element could not sound multiple audio tracks at once (and didn't reliably play alongside the video). Replace it with a pool of <audio> elements — one per audio track, created on demand and slaved to the playhead — so every audio track plays in parallel with the video's own soundtrack, mirroring the multi-source amix that export produces. - audioSegments now carries trackId; useEdlPlayback groups by track and drives one element each (src/seek/volume/play/pause), pausing tracks with no covering segment; pool torn down on unmount - PreviewMonitor no longer owns the <audio> element (managed in the composable) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Create per-track preview audio via document.createElement + appendChild (hidden) instead of a detached new Audio() — Chromium plays in-document media more reliably; removed on unmount. Verified live: with a video on V1 and music on A1, both the video element and the A1 audio element play in parallel and in sync (currentTime lockstep within 0.01s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rame
The per-track audio elements were slaved to the video-derived playhead and
re-seeked whenever drift exceeded 0.15s — which happened continuously for real
media, stalling the decoder so only fragments ("beats") leaked through while
the video played smoothly. Now each element hard-seeks ONLY on segment (re)entry,
an explicit scrub/seek, or large drift (>0.5s); a plain playhead advance during
playback no longer touches currentTime.
Verified live: over a 3s play window the A1 music element fired 1 seeking event
(the initial anchor) and 0 stalls, advancing 2.9s — smooth continuous playback
in parallel with the video, vs. the prior per-frame re-seek stutter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e sound)
syncAudio compared el.src (read back) against seg.src to decide whether to
(re)load. But reading el.src back returns a normalized/encoded URL
(media:///Users → media://users, %20 for spaces), so the comparison was ALWAYS
true — the element's .src was reset every frame, firing hundreds of
emptied/loadstart reloads per second and never reaching a playable readyState.
Result: audio played in Source preview (Vue binds :src once) but was silent on
the timeline. Track the intended src on el.dataset.edlSrc and set .src only when
that raw value changes.
Verified live with a spaced filename ("Dast man nist - shadmehr.mp3"), V1 muted:
emptied/loadstart 409→0, readyState 1→4, plays continuously.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…racks - Shrink the Filmstrip/Context view toggle (py-1→py-0.5, smaller icons) so it matches the Source/Timeline switcher instead of towering over the toolbar. - The track lanes had overflow-y-hidden, so V1/A1/OV clipped once they exceeded the panel height. Enable vertical scroll on the lanes, pin the ruler (sticky top), and mirror the scroll onto the header column via transform so headers stay aligned with their lanes. Verified live: 7 tracks → scrollHeight 408 > clientHeight 137, scrollable, and after scrollTop=60 the header column transform is translateY(-60px) (synced). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the mutually-exclusive Chat/Inspect/Revisions tabs (which hid a selected clip's details while chatting) with a split rail: the top pane toggles Inspect | Revisions, and Chat is always visible in the bottom pane. Selecting a clip while browsing Revisions flips the top pane to the Inspector; Chat stays put. (Supersedes the reverted separate-window approach.) Verified live: with a clip selected, the Inspector props (Track/Start/Source In) and the Chat input render simultaneously. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split the right rail into two closeable rows: [Chat | Revisions] on top and [Inspector] below, each with its own close (✕) and a reopen strip. Closing a single row gives its space to the other; closing BOTH narrows the rail to a slim icon strip so the preview player widens. Row open/closed state persists. Selecting a clip auto-opens the Inspector row. Verified live: player column 532px (both open) → 828px (both closed) → 532px on reopen; closing one row leaves width unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per feedback, [Chat | Revisions] and [Inspector] now sit side by side as two independently-collapsible COLUMNS (was stacked rows). Each has its own close (✕) and collapses to a slim vertical strip (reused CollapsedRail) to reopen. Closing a column frees its horizontal space so the preview player widens directly; column open/closed state persists. Verified live: player column 280px (both open) → 558px (Chat closed) → 816px (both closed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Chat/Inspector columns used an ✕ to close while the Media panel uses a sidebar-collapse icon. Swap the ✕ for the mirror tabler--layout-sidebar-right-collapse (same size + "Collapse panel" tooltip) so the collapse/expand pattern is consistent across all three panels (the reopen strips already use the matching sidebar-right-expand). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Timeline Video Editor — audio-kind assets + panel UX
This branch brings the FrameFlow timeline video editor (the CapCut/Premiere-style surface described in
video-editor-prd.md) to feature-complete for its v1 scope, closing out the last v1 item — audio-kind assets — and polishing the editor's panel layout. It is a large feature branch (the editor was developed here on top ofmain); the sections below group the diff by capability, with the most recent work described in most detail.🎧 Audio-kind assets (the headline — last remaining v1 item)
Imported audio files (local and URL/YouTube) are now first-class:
MediaAsset.kinddetected on import; metadata probe tolerant of audio-only files.segmentpreprocess step (silence split + fixed-interval fallback) so audio — including music with no silence — is divided into contiguous, placeable pieces without needing Gemini. PuresegmentAudio()unit-checked across music/speech/all-silence/tiny inputs.<audio>element, in parallel and in sync with the video's soundtrack. Two follow-up correctness fixes: elements free-run instead of re-seeking every frame (was stuttering to "beats"), and the per-frame.srcreset (frommedia://URL normalization on readback) that left the timeline silent while Source preview worked.render.tsslices regions at the union of video and audio boundaries and mixes per region with a k-inputamixgraph (gain, mutes, gaps, retime preserved); ffmpeg-validated.🖥️ Editor panel UX
[Chat | Revisions]and[Inspector]— so a selected clip's details and the chat are visible at once. Closing a column frees horizontal space and widens the player; state persists.🧱 Underlying editor (developed on this branch)
The diff also contains the editor foundation this work builds on: media import + per-asset preprocessing, the purpose-built multi-track timeline (drag/trim/split/ripple/retime/snap/zoom/markers), EDL preview, AI prompt bar + personas, item-id-keyed
TimelineDiff, the branchable revision tree, and the segment-then-concat export engine.Verification
npm run build+ typecheck clean (remaining vue-tsc errors are pre-existing innode_modules/pilotuiand unrelated app files).audio, 30s music → 2 auto-segments, drag-to-A1, parallel in-sync playback (video + A1currentTimelockstep, 0 stray seeks/reloads), and the panel layout (player width 280→558→816px as columns close).segmentAudio()and the region-slicing logic exercised with unit-style checks.Notes for reviewers
originalPath— no proxy), not a live composite; the final composite happens only at export.MediaKind,gain,muted,preservePitchalready existed in@shared/types.🤖 Generated with Claude Code