feat: timeline video editor (v1 + completion + UX pass) - #56
Merged
Conversation
Enhance pipeline with experimental mode, automated yt-dlp setup, and refactor image handling
Integrate auto-use images toggle across chat interface and pipeline flow and Enhance VideoNode with videoStore injection and
Implements the first two milestones of the timeline video editor PRD
(video-editor-prd.md):
M0 — Scaffolding
- Thread.type 'editor' + EditorDocument data model in shared/types.ts
(MediaAsset, Clip, Track, TimelineItem incl. speed/retime, persona &
TimelineDiff stubs)
- ThreadManager.createEditorThread (no chat preprocessing auto-start) and
updateThreadWith queued mutator for race-free concurrent doc writes
- /editor/:id route, Home "Video Editor" card, type-based thread routing
- Full-bleed 4-zone editor shell (media / preview / inspector / timeline)
with editorStore (ownership-split autosave vs thread-updated echoes)
M1 — Media import + per-asset preprocessing + selectable pieces
- src/main/editor: per-asset orchestrator (proxy -> scenes -> clips ->
progressive thumbnails), K=3 concurrency cap, per-asset abort, task ids
namespaced `${assetId}:step`, opt-in Gemini scene descriptions reusing
extraction.generateSceneDescription via an asset-scoped context
- IPC: create-editor-project, save-editor-doc, add-media-asset,
import-media-url (per-asset progress), remove-media-asset,
preprocess-media
- Media panel with live per-step progress, error isolation + retry,
interrupted-run resume; clip tray with selectable scene pieces;
preview monitor with clip in/out playback; inspector
- Editor-aware repairThreadPaths; scenedetect threshold param
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements PRD milestone M2 — the editor now edits:
Timeline & manual tools
- Purpose-built multi-track timeline (pxPerSecond model, shared scroll
container for ruler+lanes, adaptive tick steps, windowed clip mounting)
- Drag pieces from the tray onto tracks (HTML5 DnD) + keyboard placement
- Pointer-gesture state machine (useTimelineInteractions): move with
snapping (item edges/playhead/markers) and cross-track validation,
edge-trim with neighbor clamping (Alt = ripple-trim), ruler scrubbing,
Escape cancels; one history step per gesture
- Split at playhead, delete vs ripple-delete (magnetic default), nudge,
markers, zoom anchored at cursor (4-400 px/s) + fit-to-window/selection,
track mute/lock/hide + add-overlay-track
- Retime/speed: constant per-clip speed 0.25-4x with numeric speed AND
target-duration fields in the inspector; on-timeline duration =
(out-in)/speed; downstream ripple; hatched visual + speed badge
- Keyboard map (space, arrows, S/Cmd+B, Delete, M, Cmd+Z/Cmd+Shift+Z,
+/-) guarded against typing contexts; ARIA roles + live seek announcements
EDL preview
- useEdlPlayback: A/B double-buffered <video> pair plays the composed
timeline — boundary switching with preload, black gap advance on wall
clock, playbackRate honors retime, item/track mute respected
- PreviewMonitor: Source | Timeline modes with auto-switching
- Filmstrip <-> Context view toggle on timeline clips (scene thumbnail
vs visual description)
Undo/redo
- src/shared/timeline.ts: pure snapshot diff engine (forward/inverse
TimelineDiff) + validated apply (schemaVersion, id/range checks,
speed clamp, duration recompute) — reused by M3 AI diffs later
- src/main/editor/history.ts: sidecar userData/editor-history/{id}.json
with 50-step ring, sparse keyframes, redo-branch truncation, atomic
writes; undo pointer persists inside the doc autosave patch so doc
and pointer stay crash-consistent; history survives restart
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements PRD milestone M3 — the AI co-editor:
Personas
- 9 built-in personas in src/main/constants/personas.ts, grouped
long-form (Podcast Editor default, Long-Form Polisher, Silence
Cleaner, Chapter Organizer, Study-Notes; targetDurationSec null =
length-preserving) and summarize (Concise, Highlight Reel,
Storyteller, Social Shorts). Built-ins are code, merged at read time,
never persisted; user personas live globally in settings.json
(get/set-personas IPC) with builtin-collision filtering
- PersonaPicker popover (grouped, clone/edit/delete) + PersonaEditorModal
(create/clone/edit, preserve-runtime checkbox, live summary line)
Prompt engine
- Constrained EditorOps schema: the model returns removeItemIds /
updateItems (whitelisted fields) / addClips (assetId + scene # refs) /
addMarkers / rationale / optional answer — never raw TimelineDiff;
opsToDiff() maps server-side (generated item ids, derived durations,
unknown refs pruned into droppedOps)
- Full context windowing (src/shared/ai-scope.ts + src/main/editor/
context.ts): selection scope -> chapter/marker window (engages past
40min/400 items, 150-item cap) -> full; per-asset scene caps with
60->30->12 degradation ladder; explicit truncation + thin-context
flags, never silent
- composeSystemInstruction: persona voice + fixed editor contract +
defaults (null duration -> PRESERVE full runtime branch)
- One structured Gemini call per turn ('editor-edit' model selection),
streamed via editor-turn-update onto persisted PromptTurn records;
abortable; usage/cost recorded to thread.usageHistory
Review UX
- Proposals render ghosted on the timeline: adds = dashed sparkle
clips, removals = struck-through, moves = ghost twins at the new
position; proposed chapter markers dashed on the ruler
- ProposalCard: rationale, op-count chips, dropped-ops/thin-context/
truncation notices, usage+cost, Accept-all / Reject
- Double validation (at receipt and at accept) so manual edits during
review never mis-apply; accept commits ONE ai-origin history step
(turnId + resultStepId stamped) — a single undo reverts the AI edit
- PromptBar: persona chip, auto-grow input, scope chip with
Selection/Chapter/Whole override, send/stop, inline error + retry;
question turns show a transient answer card
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements PRD milestone M4 — the editor now exports real video files, completing the v1 loop (import -> split -> edit -> export): Render engine (src/main/editor/render.ts) - Fast path: single-source, speed-1, unmuted, gapless timelines map to TimelineSegment[] (toFixed(3) — String() exponent forms would break timeToSeconds) and reuse assembleVideo unchanged - Region path (PRD option C, segment-then-concat): the sequence slices into clip/gap regions; each renders a UNIFORM mp4 intermediate (h264 videotoolbox 8M on mac / libx264 crf18 elsewhere + aac 48kHz stereo, -video_track_timescale 90000, both A+V streams always present) normalized to source-derived WxH/FPS (aspect-preserving pad — NOT the blindly-seeded timelineMeta); stitched via concat demuxer -c copy +faststart with a one-shot re-encode fallback - Retime: setpts=(PTS-STARTPTS)/speed + chained atempo (0.5-2.0 per stage; asetrate when preservePitch=false); apad + output -t pin A/V to exact region duration - Gaps render black+silence (color/anullsrc lavfi); muted items/tracks and audio-less sources get anullsrc so concat stream layout never varies; input-side -ss/-t seeking for speed with frame accuracy - Pre-flight: regions computed + all source files existence-checked synchronously — bad exports reject the invoke, no ghost renders - Duration-weighted progress (regions 0-96, stitch 96-100) over new editor-render-progress event; abort kills ffmpeg + cleans workdir; renders auto-abort on project delete Overlap hardening (bug found in verification) - AI-accept placements weren't gesture-clamped and could overlap items: new repairOverlaps() (shared/timeline.ts) pushes later items right until sequential; runs on proposal apply AND on project load (heals existing docs); export self-repairs by butting instead of throwing; left-edge ripple-trim now also clamps against its left neighbor Export UX - ExportDialog: quality choice (original / 480p proxy preview) -> live progress + cancel -> done (Save As... via save-video, Open project folder) / neutral canceled state; closing the dialog does NOT abort — render state lives in the store - Header Export button with live "Exporting N%" label; disabled only when the timeline is empty; overlay/text-track notice line Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…al flow
Replaces the M3 ghost-preview + Accept/Reject flow with a branchable
revision tree, mirroring the chat editor's every-generation-is-a-version
paradigm:
Prompt flow rework
- An AI prompt result now APPLIES IMMEDIATELY (validated + overlap-
repaired) and lands as a new revision branching from the current one;
the timeline switches to it. AiResultCard shows "Applied - V{n}" with
rationale/counts/cost/notices and a "Back to V{parent}" jump.
Rejecting = switching back; the result stays in the tree.
- Unsaved manual work is silently auto-checkpointed BEFORE the AI
revision is created, so the AI revision's parent snapshot is exactly
the state the diff applied to and hand edits are never stranded.
- Cmd+Z still undoes an AI diff in place (ring step preserved); the
fine-grained ring resets only on revision SWITCHES (stale diffs would
corrupt a switched snapshot via redo) — new clear-editor-history IPC.
- Removed: pendingProposal state, ghost computeds/props/styling in
TimelineClip/TrackLane/TimelineRuler, ProposalCard, accept/reject.
Revision tree core
- EditorRevision {id, parentId, seq, origin init|ai|manual, label,
turnId/personaId, FULL snapshot incl. markers} in a new sidecar
userData/editor-revisions/{threadId}.json (temp-then-rename writes,
own monotonic revisionCounter so V numbers stay clean and never
reused, cap 100 with oldest-LEAF-only pruning, root protected).
Snapshots are self-contained by design — never diff-chained into the
capped undo ring. Doc carries only currentRevisionId.
- Lazy root bootstrap ("Original") on first checkpoint/prompt — zero
migration; sidecar loss never mutates the working doc.
- switchRevision: dirty guard (Save & switch / Discard / Cancel),
snapshot restore incl. markers, ring reset, selection prune, playhead
clamp. Subtree delete via recursive parent-pointer collect (root
undeletable; current relocated to the subtree's parent first).
Switch/delete blocked while a prompt runs (parentage correctness).
UI
- Right rail now tabbed Inspect | Revisions (badge dot when an AI
revision lands). RevisionsPanel: DFS tree list with depth indent,
V{n} pills, origin icons (persona emoji / bookmark / flag), first-
clip thumbnails, relative times, current ring + dirty asterisk,
hover subtree-delete. SaveRevisionButton with inline label popover
("No changes since V{n}" when clean) + timeline-toolbar bookmark.
- RevisionGraphModal: standalone Vue Flow embed (isolated instance) —
tidy tree layout (main line down, branches fan right, leaf-column
assignment), revision cards with 3-thumb strips, click-to-switch
with the modal staying open for branch-hopping, fitView on open.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Register @playwright/mcp (.mcp.json) attaching over CDP at :9222 so the app's Vue UI can be driven and screenshotted for validating changes. Add `start:debug` script to build and launch Electron with the remote debugging port open. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… editor PRD Adds the §0 status ledger (M0–M4 + revisions shipped, per-milestone commits, not-yet-built list), rewrites §5.7 to the apply-immediately revision flow (§5.7a), and updates milestones/traceability to match what shipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pletion) Phase 1 — transcript → clip.text (PRD §5.2): - New opt-in 'audio' + 'transcript' preprocess steps reusing the chat extraction phases verbatim through the asset-scoped PipelineContext - Transcript excerpts merged into each Clip.text by [in,out] time overlap, preserved across scene re-runs; enriches AI context beyond visual-only - Inspector asset mode gains a "Transcribe (uses Gemini)" opt-in button; the existing clip-mode Transcript block now populates Phase 2 — silence finder (PRD §5.6, assistive/review-only): - detectSilence() ffmpeg helper (silencedetect stderr parsing) + shared SilenceRegion type + find-silence IPC channel - SilenceFinder.vue: tunable noise-floor/min-duration thresholds, scan, reviewable region list with seek-on-click, explicit ripple-delete apply - Scan state lives in editorStore keyed by asset, so results stay visible across Inspector modes (asset/clip/item) while previewing pieces - applySilenceRegions carves regions out of placed items per track, clamped to item bounds, closes gaps, single undoable history step Verified end-to-end in the app against a synthetic speech/silence video: detected regions match ffmpeg ground truth, ripple math exact (27.44s → 17.37s), undo restores across restart, transcription windowed correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ions PRD §5.2 corrective controls (editor v1 completion, phase 3): - ClipTray header gains merge-selected / split-selected buttons and a collapsible detector panel (threshold slider 5–60 + Re-detect) that re-runs detection via the already-wired preprocess-media threshold param - mergeClips/splitClip main-process mutations (clips are main-owned): adjacency-validated merge with joined visual/text, midpoint split with min-piece guard; both reindex and clear masterSegmentIndex - Validation runs before the queued mutator so rejections reach the renderer (updateThreadWith swallows mutator throws); the mutator re-checks and skips on mid-flight changes - Inline tray error surfaces rejected operations Verified live: merge [0,5]+[5,10]→[0,10] with joined transcript, split back to halves, non-adjacent merge rejected with visible message, re-detect rebuilds pieces and preserves epsilon-matched transcripts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…vigation) PRD §5.3 (editor v1 completion, phase 4): - TimelineMinimap: always-visible overview strip under the toolbar mapping the full sequence — per-track item blocks, marker ticks, playhead line, and a visible-window rectangle; click/drag scrolls the main timeline (navigation only, never moves the playhead) - Toolbar gains a Markers & chapters popover: sorted marker list with label + timecode, jump-on-click (seek + scroll into view), per-row remove, and a press-M empty-state hint Verified live: minimap window rect tracks zoom/scroll, 90% strip click jumps scrollLeft proportionally, marker jump lands the playhead at the marker time, markers persist across autosave. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRD §5.11 (editor v1 completion, phase 6): a global media query disables fade/pulse/shimmer animations, hover lifts, press scales, and smooth scrolling when the OS requests reduced motion; spinners slow instead of stopping (they convey busy-state), and functional motion (playback, the playhead, progress widths) is untouched. scroll-behavior needs !important because a bundled third-party sheet re-declares it after our block. Verified via emulated reducedMotion in both directions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRD §5.5 (editor v1 completion, phase 5): - generateFilmstrip(): ONE ffmpeg pass (fps=1/interval, 120px) replaces the process-per-frame extractFrame for strips; interval = max(1s, dur/300) so a multi-hour source stays a single bounded run - New 'filmstrip' preprocess step (reuses the preprocess-media channel and the K=3 heavy-slot cap) populating the previously dormant MediaAsset.filmstrip; cached per asset under frames/strip/ - editorStore lazily requests strips when filmstrip view is active for completed assets placed on the timeline - TimelineClip tiles one lazy-loaded frame per ~64px slot across the clip's source range (zoom-adaptive density, ≤60/clip), falling back to the scene thumbnail until the strip lands Verified live: 27 frames auto-generated on view, clips tile floor(w/64) frames at every zoom (87px→1, 301px→4). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Six of seven deferred items shipped and verified (transcript, silence finder, sensitivity + merge/split, minimap + jump list, dense filmstrip, reduced-motion); audio-kind assets remain, scoped as its own milestone with the render-engine region-slicing prerequisite spelled out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TimelineClip's hovering flag was set on pointerdown but never cleared — no pointerenter/pointerleave existed — so once a clip had ever been clicked its edge trim handles rendered permanently, visually swallowing narrow clips. Wire real hover tracking and drop the pointerdown hack. Verified live: handles absent on untouched clips, present while selected/hovered, gone after deselect once the pointer leaves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Popover and playhead were both z-30, and the playhead sits later in the DOM, so it painted through the list. Raise the popover to z-50 (backdrop z-40), above every in-panel layer (clips z-10/20, snap guide z-20, playhead z-30). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every long-running step now reports percent or count instead of an
indeterminate spinner:
- scenes: parse PySceneDetect's tqdm 'Progress: N%' from streamed stderr
(emitted even without a tty) into the task bar
- audio + descriptions: derive percent from the reused phases' status
strings ('Converting… 45%', 'Analyzing scenes 51 to 100 / 418')
- transcript: honest stage milestones (10 upload/transcribe → 85 merge →
100) since a single Gemini call has no incremental signal
- filmstrip + silence scan: ffmpeg progress events; silence streams
'editor-silence-progress' over IPC and the scan button shows 'Scanning… N%'
- progress writes throttled to ≥5% (task) / ≥2% (event) steps since every
task update persists the thread JSON and broadcasts
Verified live on a 3-minute import: scenes streamed 0→100 in ~10 steps,
audio 0→48→100, transcript 10→85→100, silence events delivered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five UX fixes in one coherent pass:
1. Transcript runs UP FRONT: DEFAULT_STEPS now proxy → audio → transcript
→ scenes → thumbnails, with audio/transcript as soft-fail steps (a
missing Gemini key or quota error falls back to scene pieces instead of
bricking the import).
2. Pieces are REAL editorial segments: clips derive from the transcript
(one piece per spoken statement + [Silence] pieces over gaps, mirroring
timeline/enrichment.ts), so the tray shows what was said, pre-populated
with text. Scene detection stays as the fallback and as the explicit
sensitivity-re-run path; thumbnails now follow clips, not scenes.json.
3. Collapsible rails: Media (left) and Chat/Inspect/Revisions (right)
collapse to slim vertical strips, persisted in localStorage; the
monitor takes the reclaimed width.
4. Header overlap fixed: Export teleports into #header-actions-portal so
it composes with the settings/theme cluster instead of colliding.
5. Prompt box → chat panel: new ChatPanel renders the full persisted turn
history (user bubbles with scope+time, answers, applied-edit summaries
with V{n}/op-count/cost chips, error+retry, running indicator) with the
restacked PromptBar pinned at the bottom of the right rail; chat is the
default tab with an unseen-activity badge.
Verified live: 3-min speech import produced 55 text-bearing pieces
(36 speech + 19 silence) with per-piece thumbnails; rails collapse/expand
(monitor 552→796px); a question turn round-tripped through the chat panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…egion The chat input stacked card-inside-card (panel border → PromptBar card → boxed persona/scope chips); flatten it to the modern chat-input shape: - PromptBar loses its own card; the panel's border-t is the only frame. Textarea on top, one quiet meta row below (persona · scope as tiny text-level controls, small send circle right) - PersonaPicker trigger: boxed chip → icon+name text control - Right-rail tabs: bordered pill container → naked segmented text tabs - AiResultCard: floating bordered card → flat tinted message block - AssetRow: border removed; hover tint, ring only when selected Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The scope control now owns the row's flexible space (flex-1 min-w-0) so its label truncates with an ellipsis before reaching the send button, which keeps a fixed ml-2 gap instead of ml-auto. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Editor dropdowns used animate-fade-in-up (fadeInUp 0.5s, translateY 10px), which reads as a slow bottom-to-up slide. Add a dedicated animate-menu (menuIn 0.12s: fade + faint 0.97→1 scale) and apply it to the persona picker, scope menu, save-revision menu, and marker jump list — each with origin-top/origin-bottom so it grows from its anchor edge. Reduced-motion neutralizes it alongside the other decorative animations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The AI needs pieces/transcript to reason about the footage, so the prompt input was inert-but-clickable before any media was ready. Add store getters hasReadyMedia (>=1 asset completed with clips) and mediaProcessing, disable the textarea + send until hasReadyMedia, and give both the input placeholder and the chat empty-state three honest states: import media → preparing (spinner) → ready. Verified live: input disabled with no media and while preprocessing, enabled once pieces exist, and a prompt then round-trips to a turn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The single-row textarea clipped the long guidance placeholders in the narrow right rail. Shorten all four to one-line hints (the chat empty-state already carries the full guidance). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Picks up the Modal fixes (empty controlled-trigger button removed, dark-theme close button no longer a light box) from the published release. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Timeline Video Editor — v1 + completion + UX pass
Adds FrameFlow's second editing surface: a CapCut/Premiere-style timeline video editor (
/editor/:id) alongside the existing AI graph editor. Full spec invideo-editor-prd.md. Branched fromdev; contains all ofdev.Shipped
v1 (M0–M4) + revision-tree rework
TimelineDiff, context windowingassembleVideofast path +assembleTimelinesegment-then-concat (retime, gaps, mutes)v1-completion round
clip.text; silence/dead-air finder (silencedetect, reviewable ripple-deletes)prefers-reduced-motiongatingUX pass
[Silence]gaps), not just scene cutsVerification
Every feature was driven end-to-end in the running Electron app via Playwright MCP against synthetic + real footage (transcript segmentation, silence math, ripple-delete, undo-across-restart, export, chat round-trips, layout/gating states).
Notes
amix) — scoped as its own milestone in the PRD.🤖 Generated with Claude Code