Collection of normalized hunspell dictionaries, used by Codebook.
This is a fork of wooorm/dictionaries that replaces the original
JavaScript/npm pipeline with a Rust tool, strips the output down to the raw
hunspell files (index.aff, index.dic, license), and validates every
dictionary with spellbook — the spell checker Codebook uses internally —
so anything checked in here is known to load.
Each dictionary lives at dictionaries/<BCP-47 code>/:
index.aff— affix file (UTF-8,SET UTF-8)index.dic— word list (UTF-8)license— the upstream license file, when the source ships one
The pipeline is one Rust binary with subcommands:
cargo run --release -- crawl # download + extract upstream sources (archive/, source/)
cargo run --release -- build # make/configure for sources that build their files
cargo run --release -- generate # decode, normalize, patch, write dictionaries/
cargo run --release -- readme # regenerate the tables in this readme
cargo run --release -- validate # parse every dictionary with spellbook
cargo run --release -- all # all of the above, in orderEvery subcommand accepts a repeatable --only <code> filter, e.g.
cargo run -- all --only uk.
A Makefile wraps the common invocations — run make help for the list.
Notes:
validateworks offline against the checked-in dictionaries — no crawl needed. Run it after any change.crawlcaches downloads inarchive/and extractions insource/(both gitignored). Delete an entry to re-fetch it.buildshells out tomakefor six sources that don't ship ready-made hunspell files (de,el,gd,nds,rw,he). Host prerequisites:makeandperl, plus theispellandhunspellCLIs forde(brew install ispell hunspell), and a C toolchain forhe.patch --only <code>applies registered patches to already-generated dictionaries in place, for adding a patch without re-crawling. Patches fail loudly when their target text is absent, so they can't double-apply.- Upstream URLs, paths, and encodings live in
src/table.rs, ported from upstream'sscript/crawl.shwith the original provenance comments.
Some dictionaries are modified after normalization, before writing. The
patches live in src/patches.rs; each asserts the exact upstream text it
expects and fails the build when upstream changes, so this list (generated
from the patch registry) is always current:
uk: remove ICONV rules that mapped every Latin letter to the digit 0, which made hunspell silently accept all Latin-script words as numbersda: escape/in three slash-containing entries (quoting is not valid dic syntax) and fix a corrupted FedEx/Fedkrog lineel-polyton: replace a tab with a space inside a REP rule (hunspell treats the tab as a field separator, breaking the rule)br: fix three entries flaggedm01— with FLAG long that is one-and-a-half flags; the defined flagm0is meantgl: fix a corrupted numeric flag2iñer30(stray text spliced into230, the present-tense suffix flag) on the entry fortumbarhy: fix theSFX VDblock header declaring 171 rules when 172 followia: remove a doubled closing bracket ([oiyu]]) from seven PFX conditionsla: fix two affix rules misspelledSFKinstead ofSFX, which also broke the surrounding block's declared rule countmn: remove the COMPOUNDRULE block — its[a0,a1,...]alternation syntax is not valid hunspell and no parser accepts itne: strip a strayXfrom 171 numeric continuation flags (17X→17); the X-less flags are the ones actually defined, matching hunspell's lenient numeric parsing; also fix three corrupted entries (a slash inside a word, a strayIin a flag list, and two lines merged into one)sa: clean up stray carriage returns inside lines: repair dic entries and affix rules where a CR is glued onto a word, and drop single-rule affix blocks whose rule has a bare CR as a whole field (dead rules hunspell could never match)tr: renumber affix flag0to9999— hunspell numeric flags are defined as 1–65000 and flag 0 is rejected by spellbook
👉 Note: preferred BCP-47 codes are used (according to Unicode CLDR). To illustrate, as American English and Brazilian Portuguese are the most common types of English and Portuguese respectively, they get the codes
enandpt.
In total 94 dictionaries are provided.
The build tool is MIT licensed (© Titus Wormer for the
original pipeline this is derived from).
Each dictionary keeps its own upstream license — see the table above and the
license file in each dictionary directory.