[DOC] hyphenation_decompounder: document no_sub_matches / no_overlapping_matches; fix only_longest_match description

[d1am0nd]

What do you want to do?

• Request a change to existing documentation
• Add new documentation
• Report a technical problem with the documentation
• Other

Tell us about your request.

Page: https://docs.opensearch.org/latest/analyzers/token-filters/hyphenation-decompounder/

Two related problems on this page:

1. Two supported parameters are undocumented. The hyphenation_decompounder
   token filter accepts no_sub_matches and no_overlapping_matches since
   OpenSearch 2.17.0 (Adding access to noSubMatches and noOverlappingMatches in Hyphenation… OpenSearch#13895), but the parameter
   table only lists only_longest_match. Suggested rows:

   • no_sub_matches (Optional, Boolean, default false): excludes subwords
     that are fully contained within another matched subword (e.g., excludes
     ball if fußball also matched). Available in OpenSearch 2.17 and later.
   • no_overlapping_matches (Optional, Boolean, default false): excludes
     subwords that overlap with another matched subword. The most restrictive
     of the three matching options. Available in OpenSearch 2.17 and later.
   • Plus a note that typically only one of the three flags is enabled, because
     a more restrictive option makes the less restrictive ones redundant.

2. The only_longest_match description is misleading. It currently reads
   "Only includes the longest subword in the generated tokens." Per the underlying
   Lucene implementation, the flag keeps the longest dictionary match per
   hyphenation start point; subwords starting at other hyphenation points are
   still emitted, so overlapping subwords still occur. It does not select a single
   longest decomposition of the whole word.

Reproduction for both points: analyzing the German input kurzehose using the
German decompounding resources from
https://github.com/uschindler/german-decompounder emits the subwords kurz, zeh,
hose — with identical output for only_longest_match: true and false,
because each match starts at a different hyphenation point. The overlapping token zeh ("toe") causes search false positives;
no_overlapping_matches: true is the setting that fixes it, but users cannot
discover it from the current docs.

I'm happy to open a PR with the table changes.

Version: 2.17 and later (parameters introduced in 2.17.0; the
only_longest_match wording issue applies to all versions)

What other resources are available?

• Feature PR in OpenSearch core: Adding access to noSubMatches and noOverlappingMatches in Hyphenation… OpenSearch#13895 (released in 2.17.0, see release notes)
• Source: https://github.com/opensearch-project/OpenSearch/blob/main/modules/analysis-common/src/main/java/org/opensearch/analysis/common/HyphenationCompoundWordTokenFilterFactory.java
• Underlying Lucene change: LUCENE-8183 / HyphenationCompoundWordTokenFilter creates overlapping tokens with onlyLongestMatch enabled [LUCENE-8183] apache/lucene#9231, shipped in Lucene 9.8.0 ("Added the abbility to get noSubMatches and noOverlappingMatches in HyphenationCompoundWordFilter")
• Lucene javadoc describing all three flags: https://lucene.apache.org/core/9_12_0/analysis/common/org/apache/lucene/analysis/compound/HyphenationCompoundWordTokenFilter.html
• Equivalent documentation in Elasticsearch for reference: https://www.elastic.co/docs/reference/text-analysis/analysis-hyp-decomp-tokenfilter
• Testing: POST /_analyze with a hyphenation_decompounder filter on input kurzehose (German patterns + dictionary), comparing output with each of the three flags enabled
• German decompounding resources used for reproduction (widely used with this
  filter; README recommends only_longest_match: true):
  https://github.com/uschindler/german-decompounder

[kolchfa-aws]

Thanks, @d1am0nd! Would you be able to put up a PR with these changes?

[d1am0nd]

Thanks, @d1am0nd! Would you be able to put up a PR with these changes?

Ah, missed the message. Will try to do later today 👍