feat: add count modes to LostInTheMiddleRanker

[kota-wilson]

Related Issues

• fixes feat: support token-based budget in LostInTheMiddleRanker #11351

Proposed Changes:

• Add a count_mode parameter to LostInTheMiddleRanker with word, char, and token modes.
• Add tokenizer_encoding for token counting with tiktoken.
• Preserve the existing word-based word_count_threshold behavior by default.
• Update tests, docs, and release notes.

How did you test it?

• hatch -e test run pytest test/components/rankers/test_lost_in_the_middle.py -q passed: 24 passed.
• hatch run fmt-check haystack/components/rankers/lost_in_the_middle.py test/components/rankers/test_lost_in_the_middle.py passed.
• hatch run test:types haystack/components/rankers/lost_in_the_middle.py passed.
• hatch run test:types test/components/rankers/test_lost_in_the_middle.py passed.
• npm run build passed in docs-website.
• pre-commit run --files ... passed.
• git diff --cached --check passed before commit.

Notes for the reviewer

The word_count_threshold parameter name is preserved for backward compatibility; the unit is selected by count_mode. Token mode follows the existing RecursiveDocumentSplitter default encoding (o200k_base).

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@kota-wilson is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[anakin87]

Hey @kota-wilson, thanks for this PR.

As indicated in #11351 (comment),

we would like to keep functionality as is for now

If you actually see the need for this change, please comment/discuss on #11351.