011 follow-up: relax ingest matching for real incoming-data patterns#146
Merged
Merged
Conversation
The author's delivered screenshots drift from source/ in ways the first run surfaced: - Duration separator is a space OR an underscore, sometimes after the minutes, sometimes after the seconds (10m_0 - 600 Hz.jpg, 7m20s_0 - 441 Hz.jpg) — split on the first [ _] instead of the first space. - Folder names (doc + gram) and image stem <-> wav matching are now case-insensitive (incoming 7m_WAV 1.jpg matches source Wav 1.wav); whitespace and token content stay exact so real drift is still reported. The copied image takes the wav's own casing, keeping each gram folder internally consistent (fixes a latent bug where the copy used the incoming stem's casing). - One publication omits the "<doc> Files" container tier and holds its gram folders directly under the doc folder; a doc folder with >= 8 subdirectories is read as that flat layout, while an in-between count (2-7) stays reported-and-skipped. Adds regression tests for every pattern (underscore separators, case-insensitive folder/stem match, wav-cased copy, case-variant ambiguity, flat publication, sub-threshold ambiguity) and updates the spec/plan-era contract, research, data-model, README docs to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H5E6YjrQ6V7d2t1otjFYiN
Contributor
|
📄 HTML preview for this PR is live:
Rebuilt on every push to this PR; removed when the PR closes. (May take a minute for GitHub Pages to refresh.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to the merged feature 011 (#145). Real author-delivered screenshots drift from
source/in ways the first pass didn't handle. This relaxes the matching engine to absorb the benign drift while still reporting genuine mismatches, and adds a container-layout variant.Changes to
scripts/ingest_gram_images.py:10m_0 - 600 Hz.jpg,7m20s_0 - 441 Hz.jpg,11m Wav 1.jpg. The token is now split on the first[ _]instead of the first space. All six real examples the maintainer listed now parse.7m_WAV 1.jpgmatches a sourceWav 1.wav. Whitespace and token content stay exact, so missing spaces and mistyped words are still reported for the operator to fix. Two case-variant screenshots for one wav collapse toambiguous.Wav 1.jpgbesideWav 1.wav), not the incoming screenshot's — fixes a latent bug where the copy used the incoming stem's case, which would have produced an inconsistentWAV 1.jpgbesideWav 1.wav.<doc> Filestier and holds its gram folders directly under the document folder. A document folder with ≥ 8 subdirectories is read as that flat layout (the doc folder is the container); a single subdirectory is the normal container; an in-between count (2–7) stays reported-and-skipped.Changes
scripts/ingest_gram_images.py— separator regex, casefolded folder/stem matching, wav-cased copy name, flat-container resolution (FLAT_DOC_MIN_GRAMS = 8)tests/test_ingest_gram_images.py— regressions for underscore separators (after minutes and after seconds), case-insensitive folder + stem match, wav-cased copy, case-variant ambiguity, flat publication, and the sub-threshold ambiguity boundaryspecs/011-import-gram-images/{spec,research,data-model,contracts/ingest-contract}.mdandREADME.md— docs updated to matchFull suite green:
python -m unittest discover tests/→ 406 tests, 1 pre-existing skip.Notes
The still-open design question from #145 is unchanged by this PR: a wav-backed
.glcthat already carries<bitmap_crop_values>is warned-and-skipped rather than relinked. Flag if the real corpus wants relink-and-preserve instead.🤖 Generated with Claude Code
https://claude.ai/code/session_01H5E6YjrQ6V7d2t1otjFYiN
Generated by Claude Code