Skip to content

[Test] Fix race in BlockRefreshUponIndexCreationIT#154991

Open
szybia wants to merge 1 commit into
elastic:mainfrom
szybia:fix-BlockRefreshUponIndexCreationIT
Open

[Test] Fix race in BlockRefreshUponIndexCreationIT#154991
szybia wants to merge 1 commit into
elastic:mainfrom
szybia:fix-BlockRefreshUponIndexCreationIT

Conversation

@szybia

@szybia szybia commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Problem

testIndexWithZeroReplicasAndAutoExpandReplicasHasClusterBlocks fails intermittently in CI with:

java.lang.AssertionError: CreateIndexResponse failed - index creation acked but not all shards were started
Expected: is <true> but: was <false>

The test holds the search shard in recovery by intercepting the search node's
TransportRegisterCommitForRecoveryAction sends, releasing exactly one, then asserting the
index becomes green.

The search node can legitimately send that registration more than once during recovery: the
first attempt can fail transiently and be retried (indexing primary not yet active ->
ShardNotFoundException; or the indexing shard has not uploaded a commit yet ->
NoShardAvailableActionException, which restarts recovery). The old test released only the
first send and left any retry blocked forever, so the search replica never started,
create-index never became shards-acknowledged, and assertAcked failed.

Fix

Replace the "release exactly one" queue with a SubscribableListener gate: every intercepted
registration subscribes to it, and releasing it forwards all held sends. Because a completed
SubscribableListener runs later subscribers inline, any subsequent retry / re-allocation send
is forwarded immediately too. The harness is now correct regardless of how many registration
sends recovery performs. Also unmutes the test.

Why I believe it is fixed

  • Reproduced the test by running continuously on VM (repro'd in 8 hours)
  • Ran the test fix continuously on a VM for ~1 day, didn't repro
  • Reproduced deterministically by injecting a retryable failure on the first registration send
    (forcing the retry the old harness dropped): failed exactly as in CI, and passed with the fix.
  • The change is a strict correctness improvement to the harness, independent of the specific
    retry trigger.

Closes #154605

@szybia szybia added >test Issues or PRs that are addressing/adding tests :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. auto-backport Automatically create backport pull requests when merged branch:9.5 labels Jul 24, 2026
@szybia
szybia requested a review from Copilot July 24, 2026 17:57
@szybia
szybia marked this pull request as ready for review July 24, 2026 17:59
@elasticsearchmachine elasticsearchmachine added the Team:Distributed Meta label for distributed team. label Jul 24, 2026
@elasticsearchmachine

Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes an intermittent failure in the stateless internal cluster test BlockRefreshUponIndexCreationIT#testIndexWithZeroReplicasAndAutoExpandReplicasHasClusterBlocks by making the transport interception harness robust to legitimate retries of TransportRegisterCommitForRecoveryAction during recovery.

Changes:

  • Replace the single-shot “release exactly one intercepted send” mechanism with a SubscribableListener gate that forwards all held (and subsequent) registration sends once released.
  • Keep the test’s recovery-blocking synchronization via CountDownLatch, but ensure recovery can’t remain stuck if the registration is retried.
  • Unmute the previously-flaky test in muted-tests.yml now that the harness no longer deadlocks on retries.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
x-pack/plugin/stateless/src/internalClusterTest/java/org/elasticsearch/xpack/stateless/recovery/BlockRefreshUponIndexCreationIT.java Updates the transport interception/release logic to forward all (and future) commit-registration sends, preventing recovery from getting stuck on retries.
muted-tests.yml Removes the mute entry for the affected test now that the underlying race is addressed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged :Distributed/Recovery Anything around constructing a new shard, either from a local or a remote source. Team:Distributed Meta label for distributed team. >test Issues or PRs that are addressing/adding tests v9.5.1 v9.6.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] BlockRefreshUponIndexCreationIT testIndexWithZeroReplicasAndAutoExpandReplicasHasClusterBlocks failing

3 participants