Text Language Identification (LID) is still an unresolved problem for most languages in the world. Significant progress has been made using methods character n-gram models (langdetect, fastText) and neural approaches (XLM-R, GlotLID), however performance is disproportionately distributed to favour high-resource languages such as English.
This project provides a CLI and a set of notebooks to benchmark existing LID models (langdetect, GlotLID, NLLB-LID, and LLMs) as well as to train your own language detector on datasets from the Mozilla Data Collective platform, such as CommonLID, Common Voice LID, Ladino and many others.
Requires Python 3.12+ and uv:
git clone https://github.com/Mozilla-Data-Collective/language-id.git
cd language-id
uv sync- Get your API key from the Mozilla Data Collective platform and add it to a
.envfile in the repo root:
MDC_API_KEY=your-api-key-here
- Evaluate a model on the CommonLID benchmark:
uv run language-id eval --eval-model glotlid --dataset commonlid --n 200- Train your own single-language detector for any language with a text corpus on MDC:
uv run language-id train --dataset your-dataset-id --lang ladNote: See our guide "bring-your-own-dataset" for details on how to parse it properly in the codebase.
Run uv run language-id --help to see all commands and options.
| Notebook | Description |
|---|---|
| run-model-evaluation | Benchmark off-the-shelf LID tools and LLMs (zero- and few-shot) side by side on CommonLID or Common Voice LID. |
| train-single-language-detector | Train a specialist LID detector for a single language and compare it against off-the-shelf baselines. |
| train-and-evaluate-local-lid | From-scratch tutorial: train a full multi-class character n-gram + Naive Bayes detector locally, evaluate it, and analyse its errors. |
| add-new-language-to-lid | Add support for a new (often low-resource) language by folding your own corpus into the training and evaluation data. |
| compare-saved-runs | Helper notebook to compare experimental results (saved runs) and create tables and graphs for further analysis. |
For more details, visit our docs
This project is released under MPL (Mozilla Public License) 2.0.