Plumb merge parallelism via `ConcurrentMergeScheduler` instead of codec by abernardi597 · Pull Request #507 · mikemccand/luceneutil

abernardi597 · 2026-01-12T16:55:12Z

By leaving the HNSW vector codec merge executor as null, the codecs will leverage the intraMergeTaskExecutor provided at merge-time to parallelize the merges.
To populate the field, the corresponding parameters are properly set using ConcurrentMergeScheduler.setMaxMergesAndThreads(int, int).

This change results in a bit of a dependency issue with constructing the codec when using the default value ConcurrentMergeScheduler.AUTO_DETECT_MERGES_AND_THREADS, as the auto-detected value only becomes available after setting it on the CMS.
I modified KnnIndexer.createIndex to take a lambda for setting non-default options in order to get the CMS and use its values to construct the codec, but it's admittedly a bit janky.

I also modified some of the reported merge timings and print statements.

These changes were originally made as part of #502.

github-actions · 2026-01-27T00:12:27Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

mikemccand

Phew, this was hard to think about. I like the motivation for this change (use Lucene's defaults, stop creating an extra executor with confusing knobs to tune), but it's too forceful (overloading CMS hard limit on merge debt and HNSW max merge concurrency).

Could we keep the HNSW merge concurrency parameter separate? Rename it to -hnswMergeWorkerCount or so, and pass that to Lucene99HnswVectorsFormat when we getCodec(...). The javadocs for that class state that you must provide and executor when numMergeWorkers > 1 but I think that's not true? It seems to (correctly) fallback to CMS's intra-merge thread pool? Maybe make upstream PR to fix that if so.

And then change what -numMergeThread does as you do here? It becomes the soft limit (maxMergeThreads) for CMS, which is also what it sizes its intra-merge thread pool to, so that will be consistent with how KnnGraphTester works today. Since you must also set CMS's hard limit on merge debt, pick something dynamic ... maybe max(numMergeThread+2, 3*numMergeThread/2) ish? And add a new parameter (-maxMergeDebt maybe?) so user could override that dynamic default if they want?

mikemccand · 2026-02-10T11:16:33Z

    numQueryVectors = 1000;
    dim = 256;
    topK = 100;
-    numMergeThread = 1;


OK we are also switching to Lucene's default concurrency with this change right? Previously knnPerfTest.py was defaulting to 1 and 1 here?

Yes, it makes more sense to me that omitting an option should use whatever the Lucene default is.
It's also more ergonomic in my opinion if you want the Lucene default; otherwise you'd specify -1 on the CLI which can be ambiguous.

mikemccand · 2026-02-10T11:32:05Z

-    numMergeThread = 1;
-    numMergeWorker = 1;
+    numMergeThread = ConcurrentMergeScheduler.AUTO_DETECT_MERGES_AND_THREADS;
+    numMergeWorker = ConcurrentMergeScheduler.AUTO_DETECT_MERGES_AND_THREADS;


OK what actually is the difference between worker and thread here, even before your change?

It looks like numMergeWorker is how much concurrency a single HNSW merge will use, and numMergeThread is the size of the globally shared (ConcurrentMergeScheduler's intraMergeExecutor) thread pool.

I don't get why these are separately configurable. If I have only one merge to run, wouldn't I want to use all available concurrency? So I should set numMergeWorker >= numMergeThread. But then it's silly to expect/ask for more concurrency than you can possibly execute, so then numMergeWorker should be <= numMergeThread. Intersecting the two ... I should just always set them to the same value?

Or, perhaps we lose throughput if we try to do too many concurrent tasks for a single HNSW merge (thread sync/context switch overhead), in which case maybe I limit workers and expect multiple merges to typically be running to soak up all the concurrent threads?

Anyway it's all intensely confusing, and this PR is great progress (eliminating extra confusion added on top by creating another executor, and falling back to Lucene's preferred default path)...

If I follow correctly, it seems like we can just set numMergeWorker to cms.getMaxThreadCount() instead of cms.getMaxMergeCount().

I'm not concerned about thread overhead since the HNSW merges are batched anyway to mitigate the overhead of over-provisioning concurrency.

mikemccand · 2026-02-10T11:46:49Z

-      ).createIndex();
+      ).createIndex(iwc -> {
+        ConcurrentMergeScheduler cms = (ConcurrentMergeScheduler) iwc.getMergeScheduler();
+        cms.setMaxMergesAndThreads(numMergeWorker, numMergeThread);


Hmmm ... this isn't quite right ... it's overloading two very different settings.

setMaxMergesAndThreads takes two parameters, maxMergeCount and maxThreadCount. Think of maxMergeCount as a hard limit on number of running merges (merge debt), and maxThreadCount as a soft limit.

When merge debt crosses the soft limit, ConcurrentMergeScheduler (CMS) begins pausing/unpausing enough merges to keep the running merge count at the soft limit. If merge debt continues to grow, CMS takes the even more drastic step of forcefully stalling the incoming indexing threads (the threads causing new segments to poof into existence, the "producers", or the "mutator threads" in GC-speak). These are CMS's two backpressure mechanisms against the adversarial mutators.

Whereas, numMergeWorkers here is a very different concept: it's how much concurrency the HNSW merger, for a single segment, is able to take advantage of. It really should not be a configurable parameter, I think -- it should be "as much concurrency as you have to offer" (numMergeThreads here).

See above, I think we can use getMaxThreadCount() for numMergeWorker. Really, -numMergeWorker should be -maxMergeCount and -numMergeThread should be -maxMergeThreads?

And add an override flag for hnswMergeThreads, perhaps.
Does it make sense to overload the flag to control whether the intra-merge executor is used or not?
That is:

if you omit the flag then the intra-merge executor is used for hnsw merges at whatever the getMaxThreadCount() is

if you provide a value for the flag then it will use a separate executor

As far as I can tell, there isn't much else that uses the intra-merge executor so it seems redundant to even support the separate executor. I wonder why this escape hatch was added instead of just using the intra-merge executor?

github-actions · 2026-02-25T00:16:21Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

abernardi597 · 2026-03-09T21:57:09Z

The latest revision of this PR:

Adds a numMaxMerge flag to optionally override the corresponding CMS parameter
Uses the existing numMergeThread flag to optionally override the corresponding CMS parameter
Uses the existing numMergeWorker flag to optionally override HNSW graph merge parallelism
- By default, use the intraMergeExecutor from CMS with parallelism set to the (possibly computed) getMaxThreadCount()
- Using -numMergeWorker 1 uses the calling merge thread (no parallelism)
- Using -numMergeWorker N for N>1 uses a separate ForkJoinPool with parallelism N

github-actions · 2026-03-24T00:13:27Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

mikemccand · 2026-04-13T12:37:00Z

Using -numMergeWorker N for N>1 uses a separate ForkJoinPool with parallelism N

Can we instead never make our own ForkJoinPool and simply pass the requested numMergeWorker to the HNSW Codec classes? I agree, this is a weird escape hatch (passing in your own ExecutorService). It may've been done for multi-tenancy (OpenSearch holding many active indexers and wanting a single shared thread pool for the HNSW intensive merging).

We can do this (limit HNSW intra-merge concurrency, yet also use the default CMS intra-merge thread pool) because the upstream javadocs on Lucene99HnswVectorsFormat look wrong:

@param numMergeWorkers number of workers (threads) that will be used when doing merge. If

larger than 1, a non-null {@link ExecutorService} must be passed as mergeExec

It looks like it will always fall back to CMS's intra-merge thread pool if you pass null for mergeExec) -- see here and here. (And maybe make upstream PR to fix? It's obviously relevant to what we are trying to do here (limit HNSW concurrency while using a single shared threadpool for all intra-merge concurrency)).

mikemccand

Phew, sorry for long delay, this stuff is really hard to think about. It requires a "perfect" Monday morning for me!

mikemccand · 2026-04-07T14:51:07Z

            throw new IllegalArgumentException("-numMergeThread should be >= 1");
          }
          break;
+        case "-numMergeWorker":


Could we name this -numVectorMergeWorker maybe? numMergeThread vs numMergeWorker is very confusing to new eyes (and even old eyes too!).

Also, this is per-merge right? Every merge will (try to) fire up this many concurrent HNSW merger threads for the one merge, each pulling BATCH_SIZE vectors (one work unit) to process, and if in aggregate all running merges now are larger than the allotted max thread count, they will run sequentially on a case by case basis? Hmm actually I have a doubt -- is this decision late binding (enough)?

Say two merges are kicking off their HNSW merging, and numMergeWorker is 4 (each merge uses 4 concurrent workers if it can), but my intra-merge executor from ConcurrentMergeScheduler allows 6 threads. (Note that merge 1 and merge 2 have already used up 2 of these threads). Say merge 1 is a bit ahead, and it creates its four workers, and kicks them off. Since the pool has 4 threads free, each of those workers is a separate thread and merge 1 gets max concurrency for its vector merge.

But merge 2 then get scheduled, creates its 4 workers, and .run() each (this method I think). But the first worker, on seeing no threads left, will run the job itself synchronously, and so merge 2 uses only one thread for its vector merging?

Even if merge 1 finishes its vector merging, frees up the threads, merge 2 is already committed to single threaded merge? Basically, you can get to a point where HNSW merging is taking a super long time (large merge) using only 1 thread when you had asked it to use N. I'll try to see if nightly benchy is tickling this... actually, InfoStream should make it obvious (HNSW merging is nicely verbose about this).

I believe you're right: merge 2 is committed to a single-threaded merge because the synchronous worker processes batches until they are all completed, then returns control to schedule the remaining workers (that all finish quickly because there is no further work).

mikemccand · 2026-04-07T14:52:16Z

+            final AtomicInteger id = new AtomicInteger(0);
+            graphMergeExecutor = new ForkJoinPool(numMergeWorker, pool -> {
+              var thread = ForkJoinPool.defaultForkJoinWorkerThreadFactory.newThread(pool);
+              thread.setName("graph-merge-" + id.getAndIncrement());


vector-merge-$id? vector-graph-merge-$id? Let's make it consistent with however we name the command line option? But make sure we get vector into the name... (this pool is only used for the concurrent HNSW merging now?).

Update: actually, let's not even make our own thread pool...

mikemccand · 2026-04-13T12:57:57Z

+        case "-numMergeWorker":
+          int numMergeWorker = Integer.parseInt(args[++iarg]);
+          if (numMergeWorker <= 0) {
+            throw new IllegalArgumentException("-numMergeWorker should be >= 1; use 1 for calling thread or omit the flag to use the intra-merge executor");


Can we reword to ; use 1 for no vector concurrency or so? More understandable to fresh eyes ... (that it is the calling thread vs some other single thread is less important?)

mikemccand · 2026-04-13T13:02:23Z

+        if (graphMergeExecutor != null) {
+          numMergeWorker = graphMergeExecutor.getParallelism();
+        }
+        log("Indexing with %d max merge(s), %d merge thread(s), and %d merge worker(s) (using intraMergeExecutor=%s)\n",


Fix the message to say %d vector merge workers?

This numWorkers is tricky because it is per-merge. Every HNSW merge will fire up this many workers?

mikemccand · 2026-04-13T13:03:58Z

+        ConcurrentMergeScheduler cms = (ConcurrentMergeScheduler) iwc.getMergeScheduler();
+        cms.setMaxMergesAndThreads(numMaxMerge, numMergeThread);
+        cms.setDefaultMaxMergesAndThreads(false);
+        int numMergeWorker = cms.getMaxThreadCount();


Put this in an else? Confusing to set it once then set it again in the if?

mikemccand · 2026-04-13T13:05:25Z

+        cms.setDefaultMaxMergesAndThreads(false);
+        int numMergeWorker = cms.getMaxThreadCount();
+        if (graphMergeExecutor != null) {
+          numMergeWorker = graphMergeExecutor.getParallelism();


Let's fixup this logic (we should never make our own ForkJoinPool) -- we allow user to limit HNSW worker count, and throw an exception of number of workers > number of CMS threads?

mikemccand · 2026-04-13T13:12:55Z

        indexTimeFilter
-      ).createIndex();
+      ).createIndex(iwc -> {
+        ConcurrentMergeScheduler cms = (ConcurrentMergeScheduler) iwc.getMergeScheduler();


Kinda ugly that we have to resort to this weird lambda/callback because of CMS dynamic defaults... is there a better way (if we improve upstream Lucene)?

github-actions · 2026-04-30T00:27:21Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

abernardi597 force-pushed the parallel-merge branch from cbf0b90 to 8ec4176 Compare January 12, 2026 17:59

github-actions Bot added the Stale label Jan 27, 2026

abernardi597 force-pushed the parallel-merge branch from 8ec4176 to 6c00719 Compare February 3, 2026 18:24

github-actions Bot removed the Stale label Feb 4, 2026

mikemccand reviewed Feb 10, 2026

View reviewed changes

github-actions Bot added Stale and removed Stale labels Feb 25, 2026

abernardi597 force-pushed the parallel-merge branch from 6c00719 to dc50806 Compare March 9, 2026 20:40

abernardi597 added 2 commits March 9, 2026 21:18

Fix merge concurrency

543294f

Add graph merge executor override

9591d33

abernardi597 force-pushed the parallel-merge branch from dc50806 to 9591d33 Compare March 9, 2026 21:18

github-actions Bot added the Stale label Mar 24, 2026

mikemccand reviewed Apr 13, 2026

View reviewed changes

github-actions Bot removed the Stale label Apr 14, 2026

github-actions Bot added the Stale label Apr 30, 2026

Conversation

abernardi597 commented Jan 12, 2026

Uh oh!

github-actions Bot commented Jan 27, 2026

Uh oh!

mikemccand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abernardi597 Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 25, 2026

Uh oh!

abernardi597 commented Mar 9, 2026

Uh oh!

github-actions Bot commented Mar 24, 2026

Uh oh!

mikemccand commented Apr 13, 2026

Uh oh!

mikemccand left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

abernardi597 Feb 27, 2026 •

edited

Loading