[BUG]: Multi-dictionary setup not working for "uk" and "en_us"

[vsevolodchernetskyi-cengage]

Issue Description

I'm bilingual and have some repositories with both English and Ukrainian words in them.

Separately (i.e., dictionaries = ["en_us"] or dictionaries = ["uk"]) spelling works as expected, but when both dictionaries are added to configuration only Ukrainian spelling is checked.

Operating System

Linux (Ubuntu/Debian)

Editor

Neovim

Codebook Version

0.3.41

Configuration

dictionaries = [
    "en_us",
    "uk",
]

# the same behavior for:
# dictionaries = [
#     "uk",
#     "en_us",
# ]

Steps to Reproduce

1. Create en_us.md with helo wordl
2. Create uk.md with привт свт
3. Run codebook-lsp lint en_us.md
4. Run codebook-lsp lint uk.md

Expected Behavior

Both files produce 2 linting errors.

Actual Behavior

Only uk.md produces 2 linting errors. en_us.md produces:

using global config /home/user/.config/codebook/codebook.toml

Out of 1 total file(s), checked 1, ignored 0, and excluded 0.
Found 0 spelling error(s) in 0 file(s).

Code Sample

Log Output

Additional Context

No response

[blopker]

Hey, thanks for the report. Looking into it, this a quirk with the uk dictionary, where every Latin-character-only word is marked as correct, which makes Codebook also mark all those words as correct, even if en thinks it's wrong. I'm not sure what a proper fix would be without special-casing uk or modifying the dictionary, however, I'm also not even sure fixing it is the right call.

Do you have mixed language files in en and uk, or would having Codebook check for uk only on specific files? With settings overrides, this can be achieved like:

# Default everywhere: English only.
# Because `uk` is NOT in the base list, it never gets a chance to
# wildcard-accept Latin words in your English files.
dictionaries = ["en"]

# Ukrainian files: swap the dictionary entirely to `uk`.
# `dictionaries =` REPLACES the list (vs `extra_dictionaries =` which appends).
[[overrides]]
paths = [
    "**/*.uk.md",        # files named like  notes.uk.md
    "docs/uk/**/*",      # everything under a docs/uk/ folder
    "translations/uk/**/*",
]
dictionaries = ["uk"]

If this better suits your use case, then maybe the fix is just better documentation.

[vsevolodchernetskyi-cengage]

hey @blopker, thanks for taking a look!

Unfortunately, I have repos with mixed languages in the same directory, even multiple use cases with same file containing both languages. So, this workaround will not work for me.

As a workaround I can just toggle dictionary whenever writing in particular language.

[the-mikedavis]

The ICONV trick in the uk dictionary is pretty clever (in a horrible kind of way 😄). It converts ASCII plus a few other chars to 0s and those are auto-accepted because numbers are always considered correct. If you can patch the dictionaries you could try changing that ICONV section in the .aff to

ICONV 2
ICONV ʼ '
ICONV ’ '

and see how that works in practice.

[vsevolodchernetskyi-cengage]

Thanks! This does fix the issue at hand for me, i.e., English starts getting linted when both dictionaries are enabled. Should I keep this patch as a local one, or do we want this to be added upstream?

[blopker]

Indeed, I was shook when I saw it, but maybe it makes sense for the original use case of checking prose? Does Ukrainian writing often have a lot of Latin words and proper names? I haven't looked to see if this is a common trick with other dictionaries... Kicking off an agent to investigate now.

As far as I can tell there are a few options (mostly for my notes, but any feedback is welcome):

1. Codebook conditionally strips these to number rules when multiple dictionaries are used. Feels dirty, but the original hack is too, so fire with fire I guess.
2. Maybe some config to Hunspell Dictionaries to not check specific 'types' for words (all Latin strings). Could just be a per-dictionary filter of some sort. This would make (planned for future) user configured dictionaries harder, or less effective than the built-in ones. This would be flexible for other issues other, unknown, dictionaries might have.
3. cSpell doesn't use Hunspell at all during runtime, instead they pre-compute a set of tries for each dictionary then merge the trees. I suspect they did this originally because they didn't have a fast/good Hunspell runtime for Typescript, but that seems to have inadvertently worked around this issue.
4. Maybe there's some configs Spellbook could add to either ignore some classes of rules. Alternatively, some extra metadata Spellbook could return to understand why a check was accepted or rejected, like ICONV rule. I don't know enough about the Hunspell format to give good guidance there though.
5. As suggested, just edit the uk dictionary. Quick and easy, but makes future uk updates harder or more error prone.

[blopker]

Ok, after a few cups of coffee and some introspection, I just modified the uk dictionary to remove the bad ICONV rules 😂

blopker/dictionaries#3

Also contains a ton of work I've been meaning to do 🤷. At some point, Codebook will pull this new version down and it should work.

[vsevolodchernetskyi-cengage]

Thank you very much for the fix!

Regarding this question:

Does Ukrainian writing often have a lot of Latin words and proper names?

Not really. It's mostly cyrillic, except maybe for some quotes here and there. So, this change feels the right way to do it.