fix: handle gemini language prediction behaviour#75
Merged
JamesMcMinn merged 4 commits intoJul 8, 2026
Conversation
dearden
approved these changes
Jul 6, 2026
dearden
left a comment
dearden
left a comment
Contributor
There was a problem hiding this comment.
This looks like a sensible fix, and very well spotted on the error - I can imagine that was a faff to discover.
This was referenced Jul 8, 2026
JamesMcMinn
added a commit
that referenced
this pull request
Jul 15, 2026
Restores the language prediction fix that was merged (PR #75) and then reverted (PR #76). The DB language-code normalization migration (21) is excluded here and will be run manually against production; app.py keeps MIGRATION_TARGET_VERSION at 20 so nothing is auto-run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes https://linear.app/fullfact/issue/AI-2211/investigate-duplicate-languages-in-pas-dropdown
Essentially, what I found when investigating was that Gemini will often not return the 2 letter ISO code for a language, but instead the 3 letter code or even sometimes the full name of the language.
This PR attempts to fix these cases in the database, and normalise the values returned by Gemini to prevent non-2 letter iso codes from making it into the database going froward.
I've tested this on a copy of the prod DB locally and it seemed to go fine with no migration issues. We might hit a few minutes of slowdown when we deploy this but I don't think that's avoidable.
There's an argument that perhaps we should instead use something like langid for this but that's a slightly bigger change.