Add +searchAfter task modifier and searchAfter benchmark coverage (#306)

[serhiy-bzhezytskyy]

Adds searchAfter (deep pagination) coverage to the search benchmarks, closing #306.

Today every sort task runs searcher.search(q, topN, sort); there's no searchAfter path. That's the blind spot #306 tracks — it's the spinoff from the Lucene 10.0 search-after regression (apache/lucene#13856), where a dynamic-pruning change (apache/lucene#13221) caused a ~10x slowdown on mostly-sorted data that no benchy caught, and was reverted (apache/lucene#13971). This adds the missing signal.

What it does

A +searchAfter=<long> task modifier, parsed and stripped like the existing +minShouldMatch / +constantScore directives:

TermDayOfYearSortSearchAfter: dayofyeardvsort//st +searchAfter=330

• TaskParser parses the modifier and builds a FieldDoc from the value. Supported for single-field int/long sorts (the numeric dynamic-pruning path); it throws for anything else.
• SearchTask threads the FieldDoc through and calls searcher.searchAfter(after, q, topN, sort) when set, else the existing search(q, topN, sort). It's folded into equals()/hashCode() so cross-run result verification and task de-duplication stay correct.

Tasks

Adds a TermDayOfYearSortSearchAfter category (5 queries, respecting the per-category cap from #226) to wikinightly.tasks, sorting on dayOfYear (int, 1..366) with +searchAfter=330 — the tail of the range, so the non-competitive prefix is exercised by dynamic pruning rather than fully collected.

A couple of things I wasn't sure about — happy to adjust:

1. I added one writeOneLine(...) in nightlyBench.py to display the new category. Add +constantScore task directive to TaskParser and msm tasks #583 (+constantScore) didn't touch that file, so I kept this as its own final commit — happy to drop it if the report populates another way.
2. I used dayOfYear because its range is easy to reason about. lastMod (long, msec) is closer to the mostly-sorted timeseries shape from #13856 — I can add a lastMod variant too if you'd prefer that (or as well). I wasn't sure of a good deep after value for it without seeing the data distribution.

Testing

Compiles against current Lucene main; ruff check --preview on the changed Python adds no new findings; the validate_nightly_task_count cap check passes for the new category; and the task line parses (modifier stripped, sort type preserved). I haven't run it through a full nightly against a real wikimedium index yet — that "first light" is where the interesting part is, so I'd love to see what it shows once it runs.

[mikemccand]

Yay first light -- I can't wait! Adding writeOneLine is great -- use that if you want to "curate" where this task is linked in the benchy index page (e.g. you want it under a category of similar queries). But that is optional -- if you don't add it, this crazy fallback loop will write it after all the curated lines.

[mikemccand]

Thank you @serhiy-bzhezytskyy -- bit by bit this is how our benchmarks improve, one battle scar at a time :) I added some minor comments.

[mikemccand]

It's kinda strange, but Lucene doesn't actually require a sort to match the after doc. A null sort (which means sort by relevance) could take any random doc in the index with a random relevance as its after, I think? I don't think Lucene validates as its collecting that when it sees that after.doc that its relevance/sort value match after... maybe it should? Anyways, let's leave this as is and require "matched after and sort" here.

[mikemccand]

Could we use Objects.deepEquals directly on the native arrays instead of converting each to List?

I'm not sure about the blast radius of making equals possibly non-trivially more costly (new objects but they do die young) -- does the benchy use this heavily -- any HashMap checks in hot spots ... probably not?

[mikemccand]

Maybe refine the wording to "We have only implemented the single-valued numeric case here (Lucene's searchAfter can handel any sort type)" or so?

[mikemccand]

Thank you for handling the two error cases separately and customizing each exception message to match each error case, and, including the failing value!

[serhiy-bzhezytskyy]

Thanks Mike — excited for first light!

One thing still open from the description (#2): want a lastMod variant too? lastMod is the article revision timestamp (epoch-msec, enwiki-20120502), so it's the high-cardinality, skewed-recent timeseries shape that's closer to what #13856 regressed on than dayOfYear's uniform 1..366. lastmodndvsort (and the skipper variant) are already in the nightly set, so it'd slot in the same way.

The one thing I'm still unsure of is a good deep +searchAfter value — the distribution isn't uniform, so I'd rather not guess. Something around 1325376000000 (2012-01-01) should sit deep in the ascending order while leaving a real competitive tail, but I'd rather pin it once first-light data shows the actual spread. Happy to add it as a follow-up commit if you'd like it.

[serhiy-bzhezytskyy]

Thanks Mike, all good points — pushed a commit addressing them:

• Objects.deepEquals (SearchTask.java): switched to it directly on the after.fields arrays, dropping the two Arrays.asList wrappers — cleaner, no per-call List allocation, and it matches how the rest of the code compares arrays in equals (e.g. Arrays.equals in SearchPerfTest / IndexState). On the cost worry: equals/hashCode are only used for cross-run task de-dup (the Map<Task,Task> tasksSeen in SearchPerfTest), not in a search hot loop, so the array walk is off the critical path.
• Comment wording (TaskParser.java): reworded to "We have only implemented the single-valued numeric case here (Lucene's searchAfter can handle any sort type)."
• On the after-doesn't-have-to-match-the-sort point: agreed, left as-is — requiring a matched after + sort is the sane thing for the benchmark.

(And thanks re: the split error cases — that one bit me once, so now I always echo the failing value.)

One still-open item from the description that crossed with your review — the lastMod variant in my previous comment: happy to add it as a follow-up if you'd like the mostly-sorted-timeseries shape too; just unsure of a good deep +searchAfter value until first-light data shows the spread.

[serhiy-bzhezytskyy]

Are there any objections to its merger? I am ready to resolve them, just let me know. Thanks

[mikemccand]

Hello sorry catching up ... +1 for lastMod searchAfter tasks. We can do a one-time task generation, enumerating histogram of actual lastMod values in the old wikipedia extract nightly benchy uses? And maybe drop comment explaining this, so we remember to regen when we upgrade corpus in the future.

Actually, there is somewhere a Python tool that I long ago used to do just this for the existing tasks. Maybe add this case to that tool?

[serhiy-bzhezytskyy]

On the lastMod tasks — I went ahead and enumerated the distribution, since a hand-picked value didn't seem safe here. Streamed all of enwiki-20120502-lines-1k-fixed-utf8-with-random-label.txt.lzma (33,332,620 docs, 0 unparseable):

min  1036457834000  2002-11-05
max  1336004363000  2012-05-03

p25  1307754154000  2011-06-11
p50  1328969443000  2012-02-11
p75  1334285269000  2012-04-13
p90  1335623810000  2012-04-28
p95  1335864378000  2012-05-01

It's very skewed: with 16 equal-width buckets over 2002→2012, the last one (2011-09-29 → 2012-05-03) holds 22,651,693 of 33,332,620 docs — 68% — and the tail back to 2002 starts at 12 docs. So a round date much before 2010 would skip almost nothing and wouldn't exercise the pruning the task exists to measure. On the dayOfYear side I used 330/366, which is roughly p90, so p90 = 1335623810000 looks like the closest analogue; p75 would be a gentler variant if you'd like both.

One caveat worth flagging: a 300k-doc prefix of the file gives p50 = 2012-04-28, three months off the true 2012-02-11 — the file isn't uniformly shuffled with respect to lastMod, so sampling isn't a shortcut here. A full pass takes about 11 minutes.

On the tool: I looked for the Python one and couldn't find it. The closest are src/python/createMinShouldMatchTasks.py (Python, but reads a static topTerms20130102.txt dump) and src/main/perf/CreateQueries.java, which is the actual task generator. Interestingly CreateQueries.java already has the same problem solved by hardcoding — IntNRQ picks its range from 0..86400 with a "Seconds in the day" comment rather than measuring the corpus. So adding a lastMod case there next to IntNRQ, with the regen comment you suggested, seems like the natural home. Which did you have in mind? I can put the enumeration in whichever you prefer.