perf: galloping search in sealed LID iterators' NextGeq - #490
Open
radmirnovii wants to merge 1 commit into
Open
Conversation
Collaborator
|
@seqbenchbot up main |
Collaborator
|
Oh-oh, @eguguchkin Something went wrong and I couldn't process your request. |
Collaborator
|
@seqbenchbot --help |
Collaborator
|
Hey, @eguguchkin Below is the help message you've requested. |
Collaborator
|
@seqbenchbot start search-aggregation |
Collaborator
|
Oh-oh, @eguguchkin Something went wrong and I couldn't process your request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NextGeqin the sealed LID iterators runs a binary search over the remainingchunk slice on every call. The AND/OR zigzag sends monotonically increasing
targets while the slice narrows from the consumption edge, so the answer is
almost always near the edge — instrumented on a 2M-doc fraction: ~50% of calls
land at distance 0 and 99.9% within 16, while binary search pays
log2(remainder) ≈ 14 probes regardless. This PR replaces it with an
exponential (galloping) search from the edge, ~2·log2(d) probes — the same
uncapped scheme as Lucene's
IntArrayDocIdSet.advanceand RoaringBitmap's
advanceUntil,which seq-db already depends on for skip masks.
The worst case is bounded at ~2× binary (2·log2(d) ≤ 2·log2(n)) and far jumps
are self-limiting (jumps over one chunk sum to at most its length): an AND
built deliberately from a skewed pair (freq ratio ≈1450, 71% of calls in the
far zone) measures at parity. A capped variant was measured and rejected — it
hurts the common zone and rescues regimes that did not occur across ~6M
instrumented calls (max distance < 128).
Benchstat, n=10, in-process
Sealed.Search, 2M-doc fraction, vsmain:(a OR b) AND cSearch results are byte-identical to
mainon a 57-query corpus (IDs andorder in both sort directions, aggregation bins, fetched docs).
Note:
fracmanager/TestCapacityExceededis red on currentmain(a73114c)independently of this change.