Skip to content

Add secondary longest-suffix exemplar guessing for Serbian #197

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nciric wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add secondary longest-suffix exemplar guessing for Serbian #197

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
74431e6
Add secondary longest-suffix exemplar guessing for Serbian
nciric Jul 20, 2026
7d75b2d
Revert manual edits to dictionary_sr.lst
nciric Jul 20, 2026
87ff785
Replace magic numbers with MAX_SUFFIX_LEN and MIN_STEM_LEN constants
nciric Jul 20, 2026
cc389e5
Replace dictionary words with unlisted words in test cases
nciric Jul 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions inflection/resources/org/unicode/inflection/exemplar/suffix_sr.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
ак,=,изузетак

@grhoten grhoten Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this file manually generated or generated from genExemplars? It seems a little short.

@grhoten grhoten Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI the other tables are mostly generated with some small manual tweaking.

@nciric nciric Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was manually generated - I'll look into genExemplars, thanks for the pointer.

њак,=,надвожњак
ац,=,дворац
ник,=,камен
ач,=,брод
ар,=,булевар
бус,=,аутобус
ан,=,Драган
ош,=,Милош
иша,=,Љубиша
а,=,Јелена
ица,=,Љубица
ија,=,Србија
ка,=,мачка
о,=,брдо
бло,=,дебло
е,=,море
сеље,=,насеље
ме,=,име
38 changes: 36 additions & 2 deletions inflection/src/inflection/grammar/synthesis/SrGrammarSynthesizer_SrDisplayFunction.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,12 @@
#include <array>
#include <iterator>
#include <memory>
#include <inflection/util/ResourceLocator.hpp>
#include <string>

static constexpr int32_t MAX_SUFFIX_LEN = 7;
static constexpr int32_t MIN_STEM_LEN = 2;

namespace inflection::grammar::synthesis {

SrGrammarSynthesizer_SrDisplayFunction::SrGrammarSynthesizer_SrDisplayFunction(const ::inflection::dialog::SemanticFeatureModel& model)
Expand All @@ -39,6 +43,7 @@ SrGrammarSynthesizer_SrDisplayFunction::SrGrammarSynthesizer_SrDisplayFunction(c
{GrammemeConstants::NUMBER_SINGULAR, GrammemeConstants::NUMBER_PLURAL},
{GrammemeConstants::GENDER_MASCULINE, GrammemeConstants::GENDER_FEMININE, GrammemeConstants::GENDER_NEUTER}
}, {}, true)
, suffixToExemplar(inflection::util::ResourceLocator::getRootForLocale(inflection::util::LocaleUtils::SERBIAN()) + u"/exemplar/suffix_" + inflection::util::LocaleUtils::SERBIAN().toString() + u".bist")
{
}

Expand Down Expand Up @@ -159,8 +164,37 @@ ::inflection::dialog::DisplayValue *SrGrammarSynthesizer_SrDisplayFunction::getD
if (dictionary.isKnownWord(displayString)) {
displayString = inflectFromDictionary(constraints, displayString);
} else if (enableInflectionGuess) {
// Let's use rule based inflection for nouns. Assume lemma is singular, nominative.
displayString = inflectWithRule(constraints, displayString);
// Primary guess: Rule based inflection for nouns.
::std::u16string ruleResult = inflectWithRule(constraints, displayString);
if (ruleResult != displayString) {
displayString = ruleResult;
} else {
// Secondary guess: Longest suffix exemplar guessing.
Comment on lines +167 to +172

@grhoten grhoten Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an interesting combination. Why not go all in for one or the other? What is the advantage of one over the other?

@nciric nciric Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rule based inflection is going to give the correct answer for words not in the dictionary (with minor exceptions).
Some of the groups are hard to implement, or require more grammatical information to do properly - the longest suffix match is replacement for those for now. I expect the results to be occasionally incorrect, so it's the least preferred approach of the 3 (dictionary, rules, suffix guess) wrt correctness, but easy to implement :).

@nciric nciric Jul 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I actually went and implemented the other 3 cases and it wasn't that bad.
I hit a snag on this one as the suffix matching would need to be hand curated to avoid some of the special cases and with the PR #198 this PR is not needed.

Let's discuss it tomorrow.

auto countString = GrammarSynthesizerUtil::getFeatureValue(constraints, numberFeature);
auto caseString = GrammarSynthesizerUtil::getFeatureValue(constraints, caseFeature);
auto genderString = GrammarSynthesizerUtil::getFeatureValue(constraints, genderFeature);
std::vector<::std::u16string> constraintsVec;
if (!countString.empty()) constraintsVec.push_back(countString);
if (!caseString.empty() && caseString != GrammemeConstants::CASE_NOMINATIVE) constraintsVec.push_back(caseString);
if (!genderString.empty()) constraintsVec.push_back(genderString);
Comment on lines +173 to +179

@grhoten grhoten Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would have expected these values to already be derived. Why are they being queried here?

Also please don't inline if statements. It makes it hard to debug, and get decent code coverage metrics.


for (int32_t suffixLen = std::min(MAX_SUFFIX_LEN, static_cast<int32_t>(displayString.length() - MIN_STEM_LEN)); suffixLen > 0; --suffixLen) {
std::u16string_view suffix = std::u16string_view(displayString).substr(displayString.length() - suffixLen);
auto exemplarResult = suffixToExemplar.findTarget(suffix);
if (exemplarResult.has_value()) {
const auto& exemplar = *exemplarResult;
int64_t exemplarGrammemes = 0;
dictionary.getCombinedBinaryType(&exemplarGrammemes, exemplar);
if (exemplarGrammemes != 0) {
auto guessedResult = dictionaryInflector.inflectExemplar(displayString, exemplar, exemplarGrammemes, constraintsVec, {});
if (guessedResult.has_value()) {
displayString = *guessedResult;
break;
}
}
}
}
}
}
return new ::inflection::dialog::DisplayValue(displayString, constraints);
}
Expand Down
2 changes: 2 additions & 0 deletions inflection/src/inflection/grammar/synthesis/SrGrammarSynthesizer_SrDisplayFunction.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
#include <inflection/dictionary/Inflector.hpp>
#include <inflection/grammar/synthesis/fwd.hpp>
#include <inflection/tokenizer/Tokenizer.hpp>
#include <inflection/grammar/BidirectionalStringMap.hpp>
#include <string>
#include <set>
#include <vector>
Expand Down Expand Up @@ -40,4 +41,5 @@ class inflection::grammar::synthesis::SrGrammarSynthesizer_SrDisplayFunction
const ::inflection::dictionary::Inflector &inflector;
const ::std::unique_ptr<::inflection::tokenizer::Tokenizer> tokenizer;
::inflection::dialog::DictionaryLookupInflector dictionaryInflector;
const ::inflection::grammar::BidirectionalStringMap suffixToExemplar;
};
29 changes: 23 additions & 6 deletions inflection/test/resources/inflection/dialog/inflection/sr.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,34 @@
<test><source case="vocative" number="singular" gender="feminine">Србија</source><result>Србијо</result></test>
<test><source case="instrumental" number="plural" gender="masculine">становник</source><result>становницима</result></test>
<test><source case="genitive" number="plural" gender="neuter" pos="adjective">плава</source><result>плавe</result></test>
<!-- Words not in the dictionary but similar in shape -->
<!-- test><source case="vocative" number="singular" gender="masculine">уранак</source><result>уранче</result></test -->
<!-- test><source case="vocative" number="singular" gender="masculine">игроказ</source><result>игрокаже</result></test -->
<!-- test><source case="vocative" number="singular" gender="masculine">пашњак</source><result>пашњаче</result></test -->
<!-- 20 test cases of words not in the dictionary with different endings (exemplar guessing & rule fallback) -->
<test><source case="genitive" number="singular" gender="masculine">уранак</source><result>уранка</result></test>
<test><source case="dative" number="singular" gender="masculine">уранак</source><result>уранку</result></test>
<test><source case="genitive" number="singular" gender="masculine">пашњак</source><result>пашњака</result></test>
<test><source case="genitive" number="singular" gender="masculine">клинац</source><result>клинца</result></test>
<test><source case="genitive" number="singular" gender="masculine">ученик</source><result>ученика</result></test>
<test><source case="genitive" number="singular" gender="masculine">играч</source><result>играча</result></test>
<test><source case="genitive" number="singular" gender="masculine">пекар</source><result>пекара</result></test>
<test><source case="genitive" number="singular" gender="masculine">микробус</source><result>микробуса</result></test>
<test><source case="genitive" number="singular" gender="masculine" pos="proper-noun">Бојан</source><result>Бојана</result></test>
<test><source case="genitive" number="singular" gender="masculine" pos="proper-noun">Радош</source><result>Радоша</result></test>
<test><source case="genitive" number="singular" gender="masculine" pos="proper-noun">Радиша</source><result>Радише</result></test>
<test><source case="genitive" number="singular" gender="feminine">кућица</source><result>кућице</result></test>
<test><source case="dative" number="singular" gender="feminine">кућица</source><result>кућици</result></test>
<test><source case="genitive" number="singular" gender="feminine">географија</source><result>географије</result></test>
<test><source case="genitive" number="singular" gender="feminine">патка</source><result>патке</result></test>
<test><source case="genitive" number="singular" gender="neuter">огледало</source><result>огледала</result></test>
<test><source case="genitive" number="singular" gender="neuter">стабло</source><result>стабла</result></test>
<test><source case="genitive" number="singular" gender="neuter">уточиште</source><result>уточишта</result></test>
<test><source case="genitive" number="singular" gender="neuter">саопштење</source><result>саопштења</result></test>
<test><source case="dative" number="singular" gender="neuter">презиме</source><result>презимену</result></test>
<!-- Rule based inflection, group 3, all nouns ending with a -->
<test><source case="instrumental">Италија</source><result>Италијом</result></test>
<test><source case="instrumental">авенија</source><result>авенијом</result></test>
<test><source case="locative" number="plural" gender="feminine">авенија</source><result>авенијама</result></test>
<test><source case="vocative">кадија</source><result>кадија</result></test>
<test><source case="vocative">кадија</source><result>кадијо</result></test>
<test><source case="vocative">уметница</source><result>уметнице</result></test>
<test><source case="vocative">птица</source><result>птица</result></test>
<test><source case="vocative">птица</source><result>птице</result></test>
<test><source case="vocative">Стана</source><result>Стано</result></test>
<test><source case="vocative">Зора</source><result>Зоро</result></test>
<test><source case="vocative">Божа</source><result>Божо</result></test>
Expand Down
Loading