fix(watcher): stop index corruption, duplicate rows, and validation bypass#1
Merged
Merged
Conversation
…ypass
Three defects on the file-watcher/index path, found by an adversarial
source probe and confirmed against current main.
Issue 2 — move corruption + own-write echo (highest risk):
- Add a self-write ledger to Store (content-hash keyed, one-shot, 5s TTL).
Every store write records before it hits disk — insert, update, the
move's write+delete, the directory-wins reconcile rewrite, nullify, and
archive — so the watcher suppresses the echo. Subscribers now fire once
per API write, not twice (or three times with reconcile). A hand-edit
that lands before the echo has a different hash and is not suppressed.
- Add a stale-remove guard: a watcher Remove deletes the index row only if
the row still points at the removed path. If a Create already moved the
row to the new path, the Remove is skipped. Keys on index state, so it is
correct regardless of event order or debounce-batch boundary. Previously
a Remove deleted the live row by file stem, dropping it from the index
while the file remained on disk.
- Emit watcher events deterministically (creates/modifies before deletes)
and treat a Create for a vanished file as a benign skip instead of a
delete-by-stem (covers rapid A->B->C re-moves).
Issue 1 — decorated stems duplicated index rows:
- Add id_for_path(), which reverse-maps a path through the collection's
path template (extract) to recover the stored {id} even from a decorated
stem like {id}-{title}.md. Route insert (determine_id), boot scan,
watcher reindex, and the delete guard through it so all id derivation
agrees. Human-readable filenames no longer create a second, divergent row
on reindex or orphan a row on delete.
Issue 3 — watcher path skipped validation and aborted the drain:
- Run the same schema validation as the API write path on watcher-ingested
documents (strict enforced); invalid docs are skipped and surfaced, not
indexed.
- Make the drain per-file fault-isolated (extracted apply_watcher_events):
a malformed file emits a ChangeEvent::Error and the batch continues,
instead of one bad file aborting the batch and dropping the remaining
drained events.
- Add ChangeEvent::Error { collection, path, kind, message } and
WatcherErrorKind { Parse, Validation, Io }; mark ChangeEvent
#[non_exhaustive] so consumers tolerate future variants.
Adds 8 regression tests (grounddb 138 -> 146; workspace 168 green):
own-write echo suppressed, hand-edit-not-eaten race, dangerous-order and
double-move keep exactly one row, decorated reindex/delete, watcher
validation rejection, and malformed-file-mid-batch keeps the good edit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This was referenced Jul 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes three defects on the file-watcher/index path, found by an adversarial source probe and confirmed against current
main. Priority order 2 > 1 > 3 (corruption > ergonomics > robustness).Issue 2 — move corruption + own-write echo (highest risk)
Store(content-hash keyed, one-shot, 5s TTL). Every store write records before disk (insert, update, move write+delete, reconcile rewrite, nullify, archive) so the watcher suppresses its echo. Subscribers fire once per API write, not 2–3×. A hand-edit that lands before the echo has a different hash and is not suppressed.Issue 1 — decorated stems duplicated index rows
id_for_path()reverse-maps a path through the template (extract) to recover the stored{id}from a decorated{id}-{title}.mdstem. Routed through insert, boot scan, watcher reindex, and the delete guard — all id derivation agrees. No more duplicate row on reindex or orphan on delete.Issue 3 — watcher path skipped validation + aborted the drain
apply_watcher_events): a malformed file emits aChangeEvent::Errorand the batch continues instead of aborting and dropping remaining events.ChangeEvent::Error { collection, path, kind, message }+WatcherErrorKind { Parse, Validation, Io };ChangeEventmarked#[non_exhaustive].Tests
+8 regression tests (grounddb 138 → 146; workspace 168 green, 0 new warnings): own-write echo suppressed, hand-edit-not-eaten race, dangerous-order & double-move keep exactly one row, decorated reindex/delete, watcher validation rejection, malformed-file-mid-batch keeps the good edit.
Reviewed upstream by liz + arabella (plan) and donovan (consumer); findings were source-traced — empirical confirmation from donovan's live worker still pending.
🤖 Generated with Claude Code