Enable nightly tests comparing classic facets with the new during-col…

[epotyom]

…lection facets

Changelist:

1. Add support for CATEGORY+SUBCATEGORY format, for nightly tests all subcategories are added to the same graph
2. Separate nightly search and facet tasks files and execution - it is required as facets require slightly different configuration to be able to compare old/new API, see details in Enable comparing main facets module agains sandbox facets implementation #325 (comment)
3. Allow missing task in candidate if benchUtil#simpleReport - required to remove facet tasks from search tasks in [2]
4. Allow missing previous results
5. Fix script name typo in README.md (not related to nightlies)

Tested locally on top of changes from this branch to enable nightly benches main...epotyom:luceneutil:fix_nightly_start_from_scratch

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[epotyom]

Results from my machine: reports.nightly.tar.gz

You can look at the links under the Faceting header to see graphs that show classic/sandbox facet API side-by-side results, for example:

[mikemccand]

Tested locally on top of changes from this branch to enable nightly benches main...epotyom:luceneutil:fix_nightly_start_from_scratch

Ha, love that branch name! Sorry...

[mikemccand]

You can look at the links under the Faceting header to see graphs that show classic/sandbox facet API side-by-side results, for example:

Holy smokes, the during-collection kicks butt! 9.2 QPS vs 3.0 QPS!? Do we validate that they produce the same results? Are they always expected to?

[mikemccand]

This is awesome -- I can't wait to run in nightly beast3 benchy.

I'm confused why you need a whole separate JVM just for the facet tasks...

[mikemccand]

Whoa, this shows that you really got a nightly benchy up and running! Because this is a back-compat layer for the first time nightly benchy runs after this is merged... phew. Thank you for the amazing attention to detail on this.

[mikemccand]

Hmm do we need vectors for this?

Oh, I see, we want this competitor to reuse the same deterministic search index, so maybe we have to match all of its config or so?

[epotyom]

Yes, exactly, just want to reuse the index

[mikemccand]

I don't quite understand -- why can't they just be additional tasks in the wikinightly.tasks and we use the one JVM to go through all tasks including "during collection facets"? I guess we'd need to enhance that baby tasks language to express/recognize this alternative facets impl?

[epotyom]

Replied below in #518 (comment). As for the tasks format, support for that was added earlier in #325.

[mikemccand]

Wow, awesome that you tested to this point and uncovered this bootstrapping problem. I've since relaxed this (removed the and DEBUG) from main I think? I was somehow hitting the same bootstrapping problem when recovering from the horrible new Linux CPU governor can't talk to old BIOS nightly benchy regression issue!

I think -reset on the command-line should let you bootstrap, but +1 to just make it all resilient to not finding past results ... hmm, that is a bit lenient, and might hide other bugs? Not sure that matters.

[epotyom]

Oh, looks like I missed that -reset fixes it. The first run requires -reset anyway, as otherwise it fails at some point later, so I think keeping this behind -reset works.

[mikemccand]

Thank you for the naming improvements!

[mikemccand]

Hmm, maybe add an example in the comment here? What exactly is a subcategory?

[epotyom]

Basically, subcategory is just a way to group graphs in the nightly test results. I figured it might be useful for something else later, so I didn't want to make it facet-specific. Hope that works!

Adding a comment in the next revision.

[mikemccand]

Ahh is subcategory for this PR/case which facets impl?

[epotyom]

Yes, basically for tasks

OrHighMedDayTaxoFacets+postCollection
OrHighMedDayTaxoFacets+duringCollection

we create a single graph for OrHighMedDayTaxoFacets with two lines - one for postCollection QPS and one for duringCollection QPS.

[mikemccand]

Oh ok so the baby tasks language is already extended to understand during_collection_facets, yay!

[mikemccand]

You can look at the links under the Faceting header to see graphs that show classic/sandbox facet API side-by-side results, for example:

Holy smokes, the during-collection kicks butt! 9.2 QPS vs 3.0 QPS!? Do we validate that they produce the same results? Are they always expected to?

Oh actually is this a tricky comparison because during-collection facets can use threads but post-collection facets does not (in luceneutil today)?

We somehow need a cleaner way for luceneutil to normalize for / communicate this in general. The wall-clock elapsed time for a task is really important since it shows how effectively the task's execution can soak up available CPU concurrency, and what your QPS would be if you had "effectively infinite" CPU cores relative to your query traffic (you are "mostly idle-ish").

The total CPU ticks (summed across all threads that worked on that task) is also important, since that dictates how much CPU oomph (total bogomips? hmm, no, not quite bogomips) you'll need in production to achieve target QPS for a given query mix at X% CPU (100% for red-line).

The existing nightly charts (I think?) are "effective QPS if you had infinite CPU cores" (1.0 / wall-clock-latency). Hmm, but we recently changed the nightly benchy to exercise Lucene's intra-query concurrency by switching to only one query in flight at a time and N CPU cores available for concurrent execution of that one query. OK I see, we still have #275 open, tricky. From that issue, and looking at TermQuery performance around June 2024, it shows how important that change was (to have nightly benchy exercise the intra-query concurrency) because we then optimize crazy bottleneck (annot GP was AtomicLong -> LongAdder) and GR (more concurrency improvements: apache/lucene#13606). You can't improve what you don't measure! Or, you do improve what you do measure?

Anyway, we must make progress on #275 ... how to best reflect these slippery measures (in charts, and in console output).

[epotyom]

Holy smokes, the during-collection kicks butt! 9.2 QPS vs 3.0 QPS!? Do we validate that they produce the same results? Are they always expected to?

We see similar QPS wins in #325, where we do compare results. However, for nightlies we don't compare during/post facet results since they are essentially different tasks in the task file. I need to think about the best way to enable comparing results across different tasks - let me know if you have any ideas!

Oh actually is this a tricky comparison because during-collection facets can use threads but post-collection facets does not (in luceneutil today)?

Correct, and not just in luceneutil, classic facets are generally sequential, with few exceptions. That said, I do expect some QPS wins since during-collection facets skip creating docIdSets for matches, so they should be somewhat faster. At the same time, there is some overhead from managing concurrency, so the final results likely depend on the task.

Some of the luceneutil result improvement (and regression for the BrowseRandomLabelTaxoFacets task — see #325 (comment)) also comes from the fact that classic facets often use dense collection even when results are sparse. The decision to use sparse vs. dense is based on the number of matching docs, but the problem is that even if all documents match, the field we're collecting facets for might only contain a small number of labels from the taxonomy index. This means we allocate huge arrays where most values are zeros. Because of that, during-collection facets always use sparse collection instead.

I'm confused why you need a whole separate JVM just for the facet tasks...

This is actually an attempt to make the single-thread vs. concurrent comparison fairer. Details are in #325 (comment), but the tl;dr is that we want to group tasks by category so that each category can maximize CPU usage, whether it uses intra- or inter+intra-query concurrency. Oh, but I just noticed that unlike runFacets.py, nightlies have numConcurrentQueries=1 , we should probably set it equal to SEARCH_CONCURRENCY for facet tasks to make the comparison fair. What do you think? I'll make the change and rerun nightlies locally. That might also explain why the QPS wins in #325 (comment) are more modest.

[epotyom]

I'm rerunning the test with numConcurrentQueries == searchConcurrency == 8, but now that I think about it, this is also unfair, since post-collection facets can use up to 8 CPUs at the same time while during-collection facets can use up to 16. So now I'm thinking we should also limit the number of CPUs available to the JVM with -XX:ActiveProcessorCount=8. Update: ActiveProcessorCount doesn't actually limit the number of CPUs, but I think we can use taskset, Docker, or similar tools instead.

The plan for the facet tests is to set all three parameters to 8 so that the benchmark tries to utilize 100% of 8 CPUs, whether through inter- or intra-query concurrency. Grouping tasks by category ensures that tasks don't steal CPU time from one another. What do you think, @mikemccand?

[epotyom]

Made these changes based on this comment: #518 (comment)

Rerunning the runFacets.py script returns:

python3 src/python/runFacets.py -source facetsWikimediumAll --cpuNum 8

  iter 19
    post_collection_facets:
      log: /workplace/epotyom/upstream_lucene/logs/facet_implementations.post_collection_facets.19 + stdout
      run: /usr/bin/taskset -c 0,1,2,3,4,5,6,7 /usr/bin/perf stat ...
      74.4 s
      22.7 actual sustained QPS; 7.5 CPU cores used
    during_collection_facets:
      log: /workplace/epotyom/upstream_lucene/logs/facet_implementations.during_collection_facets.19 + stdout
      run: /usr/bin/taskset -c 0,1,2,3,4,5,6,7 ...
      140.6 s
      11.8 actual sustained QPS; 7.3 CPU cores used
Report after iter 19:
||Task ('BrowseDateTaxoFacets', None)||P50 Base 353.452528||P50 Cmp 444.704588||Pct Diff 25.817345406000328||P90 Base 534.788849||P90 Cmp 513.124567||Pct Diff -4.[REDACTED:CREDIT_DEBIT_NUMBER]||P99 Base 600.248772||P99 Cmp 644.1296||Pct Diff 7.310440278585017||P999 Base 604.823993||P999 Cmp 658.119915||Pct Diff 8.811806842457724||P100 Base 976.356843||P100 Cmp 665.555239||Pct Diff -31.832788004539033
||Task ('OrHighMedDayTaxoFacets', None)||P50 Base 545.207263||P50 Cmp 375.693699||Pct Diff -31.091582138369283||P90 Base 652.38529||P90 Cmp 402.635783||Pct Diff -38.28251660916512||P99 Base 749.069611||P99 Cmp 445.77733||Pct Diff -40.489198406421536||P999 Base 776.462811||P999 Cmp 508.015391||Pct Diff -34.573120076963995||P100 Base 935.150558||P100 Cmp 544.14523||Pct Diff -41.81201889418111
||Task ('BrowseMonthTaxoFacets', None)||P50 Base 483.050314||P50 Cmp 553.825946||Pct Diff 14.651813682497686||P90 Base 659.272807||P90 Cmp 629.043748||Pct Diff -4.5852124763883815||P99 Base 731.551274||P99 Cmp 695.572694||Pct Diff -4.918121433002942||P999 Base 731.758479||P999 Cmp 704.71417||Pct Diff -3.6957971483921814||P100 Base 732.360668||P100 Cmp 723.489077||Pct Diff -1.[REDACTED:CREDIT_DEBIT_NUMBER]
||Task ('AndHighMedDayTaxoFacets', None)||P50 Base 20.970577||P50 Cmp 16.918945||Pct Diff -19.32055565280821||P90 Base 36.609642||P90 Cmp 23.000297||Pct Diff -37.17420946099391||P99 Base 58.0216||P99 Cmp 34.950881||Pct Diff -39.762293697519546||P999 Base 72.961417||P999 Cmp 36.628383||Pct Diff -49.79759918862322||P100 Base 76.604544||P100 Cmp 49.319871||Pct Diff -35.61756467083729
||Task ('MedTermDayTaxoFacets', None)||P50 Base 57.294697||P50 Cmp 36.036594||Pct Diff -37.10309001197789||P90 Base 90.878953||P90 Cmp 57.730434||Pct Diff -36.47546313611249||P99 Base 223.466017||P99 Cmp 216.889888||Pct Diff -2.942787045781543||P999 Base 227.444359||P999 Cmp 242.238155||Pct Diff 6.[REDACTED:CREDIT_DEBIT_NUMBER]||P100 Base 397.845082||P100 Cmp 330.352234||Pct Diff -16.96460533349008
||Task ('AndHighHighDayTaxoFacets', None)||P50 Base 138.834646||P50 Cmp 100.492279||Pct Diff -27.61729013952324||P90 Base 168.474874||P90 Cmp 123.103071||Pct Diff -26.930901874424315||P99 Base 234.61927||P99 Cmp 152.185084||Pct Diff -35.135300693758026||P999 Base 259.068219||P999 Cmp 236.38227||Pct Diff -8.75674719483828||P100 Base 427.900727||P100 Cmp 280.833183||Pct Diff -34.36954758901356
||Task ('BrowseDayOfYearTaxoFacets', None)||P50 Base 281.829858||P50 Cmp 446.900065||Pct Diff 58.57087257234468||P90 Base 432.140499||P90 Cmp 540.965774||Pct Diff 25.18284568371363||P99 Base 525.544402||P99 Cmp 809.468948||Pct Diff 54.024844507810016||P999 Base 537.826243||P999 Cmp 859.206043||Pct Diff 59.755321385460924||P100 Base 541.182258||P100 Cmp 880.403615||Pct Diff 62.68153694720714
||Task ('BrowseRandomLabelTaxoFacets', None)||P50 Base 479.975726||P50 Cmp 1399.787906||Pct Diff 191.63722875435582||P90 Base 718.212118||P90 Cmp 1664.940973||Pct Diff 131.81744379868564||P99 Base 1003.258606||P99 Cmp 1821.479143||Pct Diff 81.55629387145272||P999 Base 1049.292285||P999 Cmp 1869.758917||Pct Diff 78.19238202061116||P100 Base 1065.933667||P100 Cmp 1883.082833||Pct Diff 76.66041436704147

                            TaskQPS post_collection_facets      StdDevQPS during_collection_facets      StdDev                Pct diff p-value
     BrowseRandomLabelTaxoFacets        2.10      (2.0%)        0.70      (0.5%)  -66.5% ( -67% -  -65%) 0.000
       BrowseDayOfYearTaxoFacets        6.35     (47.1%)        2.34      (0.8%)  -63.2% ( -75% -  -28%) 0.000
           BrowseMonthTaxoFacets        2.60     (12.3%)        1.85      (4.0%)  -28.9% ( -40% -  -14%) 0.000
            BrowseDateTaxoFacets        2.82      (1.3%)        2.16      (5.1%)  -23.4% ( -29% -  -17%) 0.000
         AndHighMedDayTaxoFacets       50.84      (4.3%)       58.34      (5.4%)   14.8% (   4% -   25%) 0.000
          OrHighMedDayTaxoFacets        2.03      (7.7%)        2.63      (1.9%)   29.1% (  18% -   41%) 0.000
        AndHighHighDayTaxoFacets        7.39      (1.4%)        9.87      (4.6%)   33.6% (  27% -   40%) 0.000
            MedTermDayTaxoFacets       19.05      (2.9%)       29.16     (11.9%)   53.1% (  37% -   69%) 0.000

This looks expected — all the Browse* tasks are match-all-docs queries, for which luceneutil uses optimized post collection facets here. We don't have this kind of optimized counting for during-collection facets yet; see this TODO comment.

All tasks that involve matching + facets perform better with during-collection facets.

[epotyom]

Nightly results match local test results above; see screenshots below. They highlight opportunities to improve during-collection facets in certain scenarios—in particular, the match-all-documents case (Browse* tasks) and dense counting (BrowseRandomLabelTaxoFacets). However, this seems outside the scope of this PR.

What is in scope:

1. Comparing results across subtasks. In the future we may have incomparable subtasks, but for now I believe we can compare them. Looking into this next.
2. Separate JVMs for search vs. facets. Facets currently need a different config (grouped tasks, limited task/searcher threads, and CPU count to ensure fair comparison). But could we use the same setup for other search tasks and enable both inter- and intra-query concurrency for all tasks? WDYT @mikemccand

[epotyom]

Ha, love that branch name! Sorry...

I originally had fix_nightly branch, but something went wrong, and I had to start over hence ...start_from_scratch :-)

[mikemccand]

Nightly results match local test results #518 (comment); see screenshots below. They highlight opportunities to improve during-collection facets in certain scenarios—in particular, the match-all-documents case (Browse* tasks) and dense counting (BrowseRandomLabelTaxoFacets).

However, this seems outside the scope of this PR.

+1

Separate JVMs for search vs. facets. Facets currently need a different config (grouped tasks, limited task/searcher threads, and CPU count to ensure fair comparison). But could we use the same setup for other search tasks and enable both inter- and intra-query concurrency for all tasks? WDYT @mikemccand

+1 -- right now in nightly I think we only test the intra-query concurrency (numConcurrentQueries=1) I think? -- it'd be great to exercise also the inter-query concurrency? But this can be separate PR? This one is already awesome progress. I'll try to make time to review this and other awesome pending PRs (@abernardi597 has good improvements pending...).

[epotyom]

Confirmed that Browse* tasks showed better QPS due to the MatchAllDocsQuery optimization. I disabled it in this commit, and now the only task that performs better with classic facets is BrowseRandomLabelTaxoFacets.

  iter 19
    post_collection_facets:
      ...
      151.1 s
      11.1 actual sustained QPS; 7.5 CPU cores used
    during_collection_facets:
      ...
      169.3 s
      9.8 actual sustained QPS; 7.4 CPU cores used

Report after iter 19:
||Task ('AndHighMedDayTaxoFacets', None)||P50 Base 18.366338||P50 Cmp 13.562355||Pct Diff -26.156455358711135||P90 Base 31.398511||P90 Cmp 20.967272||Pct Diff -33.222081773240774||P99 Base 76.168848||P99 Cmp 37.070021||Pct Diff -51.331781990453635||P999 Base 77.150165||P999 Cmp 37.949125||Pct Diff -50.811349528546046||P100 Base 111.220091||P100 Cmp 42.869246||Pct Diff -61.4554837938408
||Task ('BrowseMonthTaxoFacets', None)||P50 Base 1156.523475||P50 Cmp 752.954232||Pct Diff -34.89503254570772||P90 Base 1410.860311||P90 Cmp 897.811946||Pct Diff -36.36422124854853||P99 Base 1741.842429||P99 Cmp 999.659685||Pct Diff -42.609063348289816||P999 Base 1743.589674||P999 Cmp 1016.352157||Pct Diff -41.70921219851179||P100 Base 1852.546622||P100 Cmp 1090.556394||Pct Diff -41.13204056248578
||Task ('BrowseDayOfYearTaxoFacets', None)||P50 Base 955.618065||P50 Cmp 591.31759||Pct Diff -38.12197449406736||P90 Base 1255.602079||P90 Cmp 735.779223||Pct Diff -41.40028634023949||P99 Base 1463.296633||P99 Cmp 1150.894419||Pct Diff -21.349206097708553||P999 Base 1485.686151||P999 Cmp 1225.918242||Pct Diff -17.48470959530403||P100 Base 1526.457641||P100 Cmp 1286.943734||Pct Diff -15.690832196502466
||Task ('OrHighMedDayTaxoFacets', None)||P50 Base 217.156333||P50 Cmp 174.162974||Pct Diff -19.798344541027042||P90 Base 269.706067||P90 Cmp 192.944467||Pct Diff -28.461206250877556||P99 Base 334.995179||P99 Cmp 337.420781||Pct Diff 0.7240707186415866||P999 Base 336.693344||P999 Cmp 340.688585||Pct Diff 1.[REDACTED:CREDIT_DEBIT_NUMBER]||P100 Base 388.746093||P100 Cmp 380.919917||Pct Diff -2.0131844772006433
||Task ('BrowseRandomLabelTaxoFacets', None)||P50 Base 1132.938774||P50 Cmp 1632.114267||Pct Diff 44.060235597515174||P90 Base 1445.902779||P90 Cmp 1849.991662||Pct Diff 27.94716829297967||P99 Base 1641.308114||P99 Cmp 2036.96571||Pct Diff 24.10623530250823||P999 Base 1694.062825||P999 Cmp 2078.187766||Pct Diff 22.67477541749374||P100 Base 1842.163096||P100 Cmp 2091.383185||Pct Diff 13.528665813637609
||Task ('AndHighHighDayTaxoFacets', None)||P50 Base 93.477532||P50 Cmp 67.642775||Pct Diff -27.637397401548853||P90 Base 115.492777||P90 Cmp 83.099867||Pct Diff -28.[REDACTED:CREDIT_DEBIT_NUMBER]||P99 Base 650.96936||P99 Cmp 106.958806||Pct Diff -83.56930255519246||P999 Base 772.754804||P999 Cmp 107.298247||Pct Diff -86.11483921619205||P100 Base 772.77134||P100 Cmp 181.988704||Pct Diff -76.44986368153872
||Task ('MedTermDayTaxoFacets', None)||P50 Base 68.638274||P50 Cmp 44.125458||Pct Diff -35.713042551157386||P90 Base 104.728277||P90 Cmp 58.742476||Pct Diff -43.90963197074272||P99 Base 170.828474||P99 Cmp 78.494652||Pct Diff -54.050603999424595||P999 Base 383.694515||P999 Cmp 170.411498||Pct Diff -55.58667342429954||P100 Base 408.005809||P100 Cmp 300.747277||Pct Diff -26.288481593652996
||Task ('BrowseDateTaxoFacets', None)||P50 Base 1028.448638||P50 Cmp 709.594422||Pct Diff -31.003416623708926||P90 Base 1282.143787||P90 Cmp 848.404895||Pct Diff -33.829192669168236||P99 Base 1535.378575||P99 Cmp 958.605513||Pct Diff -37.56552757680626||P999 Base 1587.1582||P999 Cmp 996.618988||Pct Diff -37.20733144307859||P100 Base 1617.287873||P100 Cmp 1019.983523||Pct Diff -36.932469473849814
                            TaskQPS post_collection_facets      StdDevQPS during_collection_facets      StdDev                Pct diff p-value
     BrowseRandomLabelTaxoFacets        0.91      (2.7%)        0.61      (1.3%)  -32.5% ( -35% -  -29%) 0.000
          OrHighMedDayTaxoFacets        4.75      (1.9%)        5.81      (2.9%)   22.5% (  17% -   27%) 0.000
         AndHighMedDayTaxoFacets       57.66      (5.1%)       71.05      (5.8%)   23.2% (  11% -   35%) 0.000
        AndHighHighDayTaxoFacets       10.68      (2.3%)       14.72      (2.7%)   37.8% (  31% -   43%) 0.000
            BrowseDateTaxoFacets        1.03      (4.7%)        1.42      (7.9%)   38.1% (  24% -   53%) 0.000
           BrowseMonthTaxoFacets        0.90      (6.2%)        1.32      (6.6%)   46.3% (  31% -   63%) 0.000
            MedTermDayTaxoFacets       15.07      (2.4%)       23.07      (5.5%)   53.1% (  44% -   62%) 0.000
       BrowseDayOfYearTaxoFacets        1.08      (2.2%)        1.71      (3.3%)   58.0% (  51% -   64%) 0.000

I added implementing MatchAllDocsQuery optimizations for the sandbox facets to the plan doc: https://docs.google.com/document/d/1PF9KWYboy6terrPp8Frizlkp1ee09RX-DsuZrBux-Oo/edit?tab=t.0#bookmark=id.l0mbts4rqrw5

I also confirmed that BrowseRandomLabelTaxoFacets performs better with classic facets because of dense counting. I tested by disabling dense counting, which made sandbox facets faster for all tasks. We already have a task for this in the plan: https://docs.google.com/document/d/1PF9KWYboy6terrPp8Frizlkp1ee09RX-DsuZrBux-Oo/edit?tab=t.0#bookmark=id.1txykoa5kvos

                            TaskQPS post_collection_facets      StdDevQPS during_collection_facets      StdDev                Pct diff p-value
         AndHighMedDayTaxoFacets       44.06      (3.9%)       55.44      (5.3%)   25.8% (  15% -   36%) 0.000
        AndHighHighDayTaxoFacets       14.07      (2.6%)       18.28      (3.5%)   29.9% (  23% -   37%) 0.000
     BrowseRandomLabelTaxoFacets        0.46      (1.6%)        0.61      (2.9%)   32.1% (  27% -   37%) 0.000
          OrHighMedDayTaxoFacets        2.12      (2.8%)        3.10      (3.0%)   46.5% (  39% -   53%) 0.000
            MedTermDayTaxoFacets       19.75      (3.1%)       29.60     (11.8%)   49.9% (  33% -   66%) 0.000
            BrowseDateTaxoFacets        0.89      (2.4%)        1.37      (8.7%)   52.7% (  40% -   65%) 0.000
           BrowseMonthTaxoFacets        0.80      (7.4%)        1.29      (8.3%)   61.0% (  42% -   82%) 0.000
       BrowseDayOfYearTaxoFacets        0.92      (1.7%)        1.70      (3.0%)   84.7% (  78% -   90%) 0.000

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[mikemccand]

Phew, this almost fell past the event horizon of my TODO list ... I'll try to page it all back in and move forward. This has some big changes to the nightly benchy right? Like separate JVM for each task or so? But mixing tasks across threads within one JVM was intentional because it forced hotspot to pay attention to N different things at once so it could not over-train/optimize to singleton task. But then we changed nightly benchy at some point to do only intra-query concurrency, so we lost that hotspot stretching.

Thank you for all the exciting work here @epotyom -- it's awesome to have this faster facets engine, and it really is important we track it in nightly benchy.

[epotyom]

Thanks for reviewing @mikemccand!

This has some big changes to the nightly benchy right? Like separate JVM for each task or so?

Right now it only starts a separate JVM for facet tasks - not a separate JVM per category. Within that JVM, facet tests are configured to:

1. Group tasks by category (groupByCat=true) - runs all tasks from one category first, then moves to the next
2. Enable both inter- and intra-query concurrency, using the same thread pool size for both
3. Limit the number of CPUs to the thread pool size, so each category does its best utilizing the same CPU budget, whether through inter- or intra-query concurrency

Without this, comparing classic facets (mostly intra-query concurrent) to during-collection facets (both inter- and intra-query concurrent) is unfair - task performance depends heavily on what other tasks are running in parallel and which thread pool they compete for.

But mixing tasks across threads within one JVM was intentional because it forced hotspot to pay attention to N different things at once so it could not over-train/optimize to singleton task.

Good point, the changes above do break this. The intent is to give the same resources to each category and compare them at their best. But since we are not creating a separate JVM per category, results can be skewed: the warmup phase in SearchPerfTest optimizes for the first category's code paths, and the second category starts cold. It may not be as bad as it sounds given the high taskRepeatCount, but it is worth fixing.

I see two options:

1. Pause this PR until Close traps when testing concurrent search #275 (Close traps when testing concurrent search) is resolved - is anyone actively working on it? If not, I can take a look.
2. Merge as-is for now and revisit when Close traps when testing concurrent search #275 is resolved.

What do you think?

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[epotyom]

I've been thinking about #275 , in particular about running tasks from different categories concurrently and measuring fair performance for each category separately. I'm not sure it is truly feasible. Even if we can count CPU utilization per category, I'm not sure we can fairly isolate IO, memory bandwidth, etc. I think that the underlying issue is that we want macro benchmark conditions (realistic concurrent load) but per-category micro benchmark results. I'm not sure we can have both at once.

Proposal: split into separate macro and micro benchmark runs, in separate JVMs.

Macro benchmarks: interleave tasks across categories as today, but report a single aggregate QPS across all categories. Interleaving is the right approach here - it prevents hotspot from over-optimizing for one code path - but extracting fair per-category results from a concurrent mixed run isn't feasible, so we don't try.

Micro benchmarks: run one category at a time so each has full access to resources and results are comparable across categories and stable day-over-day. Concretely:

• Tasks from different categories don't run concurrently - one category fully completes before the next starts.
• Set searchConcurrency, numConcurrentQueries, and available CPUs (via taskset) to the same value, so that each category uses the same total CPU budget regardless of whether it achieves that through inter- or intra-query concurrency. This is important for fairly comparing during-collection facets (intra-query parallel) against other implementations.
• Use a single JVM for all categories. The ideal would be a separate JVM per category to eliminate JIT ordering effects entirely, but at 82 categories in the nightly tasks file × 20 JVMs that's 1640 JVM launches - sounds too expensive. Instead, randomly shuffle category order across JVM runs to average out JIT ordering bias and increase taskRepeatCount to ensure each category runs enough iterations to saturate all available CPUs within its window.

WDYT @mikemccand ?

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!