Rust pipeline - #3
Merged
Merged
Conversation
The upstream affix file used ICONV rules to map every Latin letter to the digit 0, making Hunspell silently accept any Latin-script word as a number. Remove those rules so Latin words are flagged as misspellings. Document this and the earlier da patch in a new "Fork modifications" section in the root readme, since the per-dictionary files are regenerated. Also remove funding.yml and ignore tsconfig.tsbuildinfo.
Replace the JavaScript/shell pipeline (crawl.sh, generate.js, npm toolchain) with a single Rust binary: crawl, build, generate, patch, validate, readme, and all subcommands, each filterable with --only. - Output per dictionary is now just index.aff, index.dic, and license; the npm packaging files (package.json, index.js, index.d.ts, readme) are removed from all dictionary dirs. - Upstream sources live as declarative data in src/table.rs, ported from crawl.sh with provenance comments. Normalization is byte-faithful to the old iconv/sed pipeline. - Fork-local fixes are a fail-loud patch registry (src/patches.rs): each patch asserts the exact upstream text it expects and errors when upstream changes. The readme patch list is generated from it. - validate parses every dictionary with spellbook (what Codebook uses), offline. This found 8 dictionaries that did not parse at all (br, gl, hy, ia, la, mn, ne, tr) due to upstream typos; all are now patched. - Six dictionaries whose upstreams disappeared are kept as frozen: fo, fur, hy, hyw, it, mk — validated but not regenerable. - CI now runs cargo test + validate. A Makefile wraps common commands.
First full regeneration with the Rust pipeline. Content drift comes from upstream branches that moved since the last crawl (mn was largely rewritten upstream; br, cy, gl, hr, ne, rw, tr changed modestly; uk only gained a trailing newline). br, gl, hy, ia, la, mn, ne, and tr previously failed to parse with spellbook (and so could not load in Codebook) due to upstream data bugs — SFK/SFT typos for SFX, wrong affix rule counts, corrupted flag lists, a doubled closing bracket, and Turkish using reserved flag 0. All are fixed by the patches in src/patches.rs; hy is frozen (dead upstream), so its fix is applied directly to the checked-in file.
lint, test, and validate, plus a check that the readme tables are in sync with src/table.rs and src/patches.rs. Crawl/build/generate need network and host tools, so they stay local. Full git history is fetched because the patch tests read their fixtures from old commits.
Codebook already references dictionaries/grc and dictionaries/sa from this repo, but they never existed here (404 at runtime). Add both: - grc from mrakia/hunspell-ancient-greek (GPL-3.0), based on the Perseus Project's Morpheus word list. Parses cleanly. - sa from Shreeshrii/hindi-hunspell (GPL-2.0). Upstream has stray carriage returns inside lines; a new patch repairs 109 dic entries and 6 affix rules where a CR is glued onto a word, and drops 119 single-rule affix blocks whose strip/condition field is a bare CR (dead rules hunspell could never match either). Both validate with spellbook and spot-check correctly with the hunspell CLI (dictionary words accepted, garbage rejected).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.