Skip to content

fix: line-wrap-split typing masquerades as an extra prompt submission#23

Merged
htlin222 merged 1 commit into
mainfrom
fix/robust-hard-segment-boundary
Jul 5, 2026
Merged

fix: line-wrap-split typing masquerades as an extra prompt submission#23
htlin222 merged 1 commit into
mainfrom
fix/robust-hard-segment-boundary

Conversation

@htlin222

@htlin222 htlin222 commented Jul 5, 2026

Copy link
Copy Markdown
Owner

The bug

A real record-session --robust demo (3 turns: a short Q&A, then a long
27-word/3-line prompt, then /theme) produced a badly desynced final video:
turn 1's (the long-prompt turn's) hard segment came out only 6.16s long — too
short to even contain its own scripted typing duration (6.74s), let alone the
real ~8.1s of Claude thinking+response that followed. Downstream, the frozen
freeze-frame chosen for the next turn's "pre" segment sourced from mid
sentence
, so a half-typed prompt sat frozen on screen for ~23s while
unrelated narration ("turn 1 done" / "now watch /theme") played over it.

Root cause (confirmed against the real capture's pixels, not guessed)

detect_anchors.signals() auto-locates the "input band" it tracks as a
FIXED, ~5-row-tall window near the bottom of the frame — deliberately tight so
response content doesn't leak in. That's fine for a short, single-line
prompt. But when a prompt is long enough to wrap onto a new line, the
input box grows a row: the just-typed line's content shifts out of that fixed
band and the freshly wrapped (still-blank) line takes its place. That's a
sharp brightness drop followed by a climb as the new line fills in —
pixel-for-pixel the same "type-then-Enter-clears" shape detect_turns uses
to mean a real submission
, even though the user never stopped typing.

Measured on the real repro: one continuous typing burst produced two
"submits" 1.52s apart (raw frames 386 and 424 @ 12.5fps), with peaks at 100%
and 93% of the frame's global max brightness — i.e. genuinely strong, not
the weaker spinner/response leakage the existing "keep the N strongest peaks"
disambiguation already knew to filter out. With 4 raw groups detected for 3
scripted turns, that logic kept both wrap pieces and discarded the turn
that actually owned the 4th (weaker, correctly-detected) group — corrupting
that turn's own submit and starving the preceding turn's done search
window (which is bounded by the next detected group's start).

Confirmed via frame-by-frame extraction (ffmpeg -ss ... -frames:v 1): at
the exact raw timestamps the old code treated as "Enter pressed" for the long
prompt, the terminal screenshot still shows the sentence being actively
typed, with a live cursor — no Enter has happened yet.

The fix

In detect_anchors.detect_turns(), fuse adjacent detected groups into one
before any n-vs-count reasoning, when all three hold:

  1. short gap (≤2s — generous vs. the ~1.5s dip measured; a real turn
    boundary always has a multi-second settle/think/PAD gap; back-to-back
    instant turns are handled by the separate, existing
    _recover_merged_instant_groups path and never show a partial dip like
    this);
  2. both flanking peaks genuinely near-max (real typed text, not moderate
    spinner leakage);
  3. the gap is RISING, not idle — a real Enter leaves the input band idle
    at its post-clear floor for the whole gap; a wrap dip instead climbs
    almost immediately because the same typing burst is still going. This
    is what actually distinguishes a wrap dip from two short-but-genuinely-
    distinct back-to-back turns (which can also have a short gap with
    near-max peaks either side) — gap-length or peak-strength alone aren't
    enough on their own.

Fixing this correctly re-surfaced a second, separate, pre-existing edge case
in this same demo: the final turn (/theme, a native menu) never shows a
real Enter-clear at all before the raw capture ends (the recording stops
while the menu is still open), so its detected group runs to the raw video's
very last analyzed frame. submit in that case just means "recording
ended here" — the existing +0.3s floor on done was pushing the segment
past the video's own reported length, which broke the freeze-frame splice
downstream (no frame exists past the file's own duration). Added a matching
clamp (with one frame of slack, mirroring BootStrategy's existing "at end"
convention in strategies.py) so done for the last turn never exceeds the
raw capture's own length.

Testing

Per CONTRIBUTING.md's testing ethos, the exact failure mode is reproduced
as synthetic pixel-array unit tests (using the real measured peak
values/gap/timings) in tests/test_detect_anchors.py, verified to fail
before the fix and pass after:

  • test_merge_same_burst_groups_fuses_wrap_split_typing
  • test_merge_same_burst_groups_leaves_distinct_turns_alone
  • test_merge_same_burst_groups_leaves_weak_spinner_leak_alone
  • test_detect_turns_wrap_split_no_longer_starves_neighbouring_turns
  • test_detect_turns_last_turn_done_never_exceeds_raw_video_length

134 pre-existing tests still pass unmodified; 139 total, all green.

I additionally used a real --robust capture exhibiting this exact bug (not
committed — see below) to diagnose the root cause via direct pixel/frame
inspection, and to verify end-to-end: re-running detect_anchors + author.py

  • splice.py against it now produces a properly-sized hard segment for the
    long-prompt turn (18.44s vs. the previous 6.16s, correctly containing the real
    submit and enough room for the actual response) and the full splice pipeline,
    which previously crashed on the newly-surfaced /theme edge case, now
    completes successfully.

On the real repro fixture

I did not commit the real capture used to diagnose this (terminal_raw.mp4 +
its capture/script/ledger/timeline siblings). The repo's .gitignore
excludes *.mp4 repo-wide save one existing hero asset, and
CONTRIBUTING.md states the unit test suite intentionally needs no external
tools (ffmpeg/vhs/tmux) to run. The synthetic tests above already
reproduce the precise pixel shape (constructed from the real measured
peak/gap/climb values), so I judged a large committed binary fixture wasn't
worth the added repo weight/precedent. Happy to reconsider if maintainers
would rather have it as a -f-added fixture.

Scope note: is this robust/tmux-specific?

Inconclusive either way from what I could test in the time available — I
did not get to build a comparable standard (non---robust, non-tmux)
capture with a similarly long, wrapping second-turn prompt to compare
against. The mechanism itself (a fixed-row input-band tracker losing a line
to box growth on wrap) is a property of signals()'s pixel-band tracking,
not of --robust/tmux capture specifically, so I'd expect it to reproduce
under standard VHS capture too given a long enough wrapping prompt — but
that's inference, not something I verified against a second real capture.
Worth confirming with a standard-mode repro if it matters for prioritization.

Issue-filing for this bug is left to the orchestrator/maintainer if still
wanted — no existing issue number is referenced here.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

…detect_turns

A long, multi-line-wrapping prompt was mis-detected as TWO (or more) separate
prompt submissions, corrupting neighbouring turns' hard-segment boundaries.

Root cause (confirmed against a real --robust capture, not guessed): the
auto-located "input band" in signals() is a FIXED, ~5-row window near the
bottom of the frame. When a long prompt wraps onto a new line, the input box
grows a row — the just-typed line's content shifts out of that fixed band and
the freshly wrapped (still-blank) line takes its place. That produces a sharp
brightness drop followed by a climb as the new line fills in — pixel-for-pixel
the same "typing fills the box, Enter clears it" shape detect_turns uses to
mean a real submission, even though the user never stopped typing.

Measured on the repro (27-word/3-line prompt): two "submits" 1.52s apart
(frames 386 and 424), both >=0.92 of the frame's peak brightness — real typed
text, not the weaker spinner/response leakage the existing GUIDED-selection
logic already knew to filter out. With 4 raw groups detected for 3 scripted
turns, "keep the 3 strongest peaks" kept BOTH wrap pieces and discarded the
turn actually mapped to the discarded group (`/theme`'s own, genuinely weaker,
native-menu peak) — corrupting that turn's submit AND starving the preceding
turn's `done` search window (bounded by the second wrap piece's start, ~1.5s
after its real submit instead of after the real ~8s think+response). The
resulting hard segment was too short to contain even the scripted typing
duration, and the freeze frame used for the next turn's "pre" segment sourced
from mid-sentence — a half-typed prompt held statically for over 20s while
unrelated narration played over it.

Fix: fuse adjacent groups into one BEFORE any n-vs-count reasoning when they
are (a) short apart (<=2s, generous vs. the ~1.5s dip measured — a real turn
boundary always has a multi-second settle/think/PAD gap), (b) both genuinely
near-max (not moderate spinner leakage), AND (c) the gap between them is
RISING rather than idle (a real Enter leaves the box idle; a wrap dip keeps
climbing because the same burst is still typing). All three conditions
matter — gap-length or peak-strength alone also fire on two short, fast,
genuinely distinct real turns.

Also clamp `done` for the video's LAST turn to never exceed the raw capture's
own length: fixing the above correctly re-surfaced a turn (in this same demo,
`/theme`) whose group runs off the very end of the analyzed video because the
recording stopped before a real Enter-clear was ever observed (an open native
menu at capture end) — previously masked by the wrap-split bug assigning it a
different, wrong group instead. Leaves one frame of slack so a degenerate
zero-length tail freeze still has a real frame to source from.

Per CONTRIBUTING.md's testing ethos, this failure mode is reproduced as
synthetic pixel-array unit tests (matching the measured real values) before
the fix, in tests/test_detect_anchors.py. The real repro artifacts (a genuine
--robust capture exhibiting this bug, plus the ledger it produced and the
hook-timestamped ground truth) were used to diagnose and verify the fix
end-to-end (author.py + splice.py rebuilt successfully; the previously
frozen half-typed frame is gone) but are not committed — the synthetic tests
already reproduce the exact pixel shape, and the repo's stated test ethos
keeps tests/ free of external tools or large binary fixtures (*.mp4 is
git-ignored repo-wide save one existing hero asset).

134 pre-existing tests still pass; 5 new tests added (139 total).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012EwptawkV4dcGhoGefzxgV
@htlin222
htlin222 merged commit 03bf57d into main Jul 5, 2026
5 checks passed
@htlin222
htlin222 deleted the fix/robust-hard-segment-boundary branch July 5, 2026 08:38
htlin222 added a commit that referenced this pull request Jul 5, 2026
#23 postmortem) (#24)

PR #23 fixed a real bug where a long wrapping prompt fooled the pixel-based
turn-boundary detector into producing a turn whose "hard" segment was
mathematically impossible (shorter than the deterministic scripted typing
time alone) — silently corrupting the ledger. It was only caught by a human
eyeballing video frames, not by the pipeline itself. This adds two
independent, complementary guards so the SAME class of bug (and others
shaped like it) gets caught automatically next time, instead of waiting for
another reactive patch.

Layer 1 (detect_anchors.py, detect_turns()): a deterministic self-consistency
floor requiring no external data. submit - typing_start must equal exactly
type_dur + pre_enter (the scripted time to type the prompt and reach Enter) —
a real submit can never be faster than that. The existing `max(0.0, ...)`
clamp used to silently absorb a violation of this; now it raises immediately,
naming the turn and the impossible numbers.

Layer 2 (author.py, main()): a best-effort cross-check of the pixel-detected
per-turn response window (done - submit) against session-timeline.jsonl's
real UserPromptSubmit->Stop wall-clock delta for the same turn, when that
file is available and its turns can be unambiguously correlated (mirrors
panel.py's existing instant-turn-aware matching). This catches a DIFFERENT
class of error Layer 1 cannot: a wrong-but-internally-self-consistent group
selection that still satisfies Layer 1's floor. It degrades silently on a
missing/malformed/uncorrelated timeline — never a hard requirement — and only
raises on a wild disagreement (tolerance: max(5.0s, 50% of the real delta),
chosen generously against detect_turns()'s own documented "many seconds"
video/wall-clock drift risk during heavy output, while still comfortably
catching a mismatch of the #23 bug's scale; validated against one real
110s-recording data point showing ~7% pixel-vs-wallclock agreement).

Layer 2 intentionally lives in author.py, not detect_anchors.py: detect_turns
already argues wall-clock timing can't be the PRIMARY source of truth (video
drift), and author.py is the orchestration layer that already knows the demo
path, per CONTRIBUTING.md's pure-logic/thin-I/O split.

12 new regression tests (3 Layer 1, 9 Layer 2); full suite green (151 passed,
up from the pre-existing 139), including every #16-#23 recovery-path
regression test unmodified.


Claude-Session: https://claude.ai/code/session_012EwptawkV4dcGhoGefzxgV

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant