[CI] InternalDistributionBwcSetupPluginFuncTest builds distribution from branches via archives extractedAssemble failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #44492 / build-logic
• elasticsearch-pull-request #166766 / build-logic

Reproduction Line:

null

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.gradle.testkit.runner.UnexpectedBuildFailure: Unexpected build execution failure in /dev/shm/bk/bk-agent-prod-gcp-1784908624290266518/elastic/elasticsearch-intake/build-tools-internal/build/tmp/integTest/junit13411486566787729469/cloned with arguments [:distribution:bwc:major1:buildBwcDarwinTar, -DtestRemoteRepo=/dev/shm/bk/bk-agent-prod-gcp-1784908624290266518/elastic/elasticsearch-intake/build-tools-internal/build/tmp/integTest/spock_InternalDistributio_sharedRemoteRepoDir14775174907064037193/remote/.git, -Dbwc.remote=origin, -Dbwc.dist.version=8.4.0-SNAPSHOT, -Dtests.bwc.mode=auto, -Dtests.bwc.dra.hash.8.x=notexist, -Dtests.bwc.dra.base.url=http://localhost:36595, --full-stacktrace, -Dtest.external=true]

Output:
> Task :buildSrc:compileJava NO-SOURCE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:processResources
> Task :buildSrc:classes
> Task :buildSrc:jar
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 9.6.
[truncated]

Issue Reasons:

• [main] 2 consecutive failures in test builds distribution from branches via archives extractedAssemble
• [main] 2 consecutive failures in step build-logic
• [main] 1 consecutive failures in pipeline elasticsearch-intake
• [main] 1 consecutive failures in pipeline elasticsearch-pull-request
• [main] 2 failures in test builds distribution from branches via archives extractedAssemble (12.5% fail rate in 16 executions)
• [main] 2 failures in step build-logic (12.5% fail rate in 16 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-core-infra.

👀

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in test builds distribution from branches via archives extractedAssemble
• [main] 2 consecutive failures in step build-logic
• [main] 1 consecutive failures in pipeline elasticsearch-intake
• [main] 1 consecutive failures in pipeline elasticsearch-pull-request
• [main] 2 failures in test builds distribution from branches via archives extractedAssemble (12.5% fail rate in 16 executions)
• [main] 2 failures in step build-logic (12.5% fail rate in 16 executions)

Build Scans:

• elasticsearch-intake #44492 / build-logic
• elasticsearch-pull-request #166766 / build-logic

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

Test Failure Analysis

Summary

The 4 InternalDistributionBwcSetupPluginFuncTest failures are caused by a transient HTTP 504 from services.gradle.org when the inner BWC source build tries to download gradle-9.6.1-bin.zip. The root cause is a gap in the test fixture's Gradle wrapper pre-population: AbstractGitAwareGradleFuncTest.setup() copies ~/.gradle/wrapper/ into gradleUserHome/wrapper/, but BwcSetupExtension invokes the inner ./gradlew with -g <gradleUserHome>-<projectName> (e.g. gradleUserHome-major1/). The setupGradleUserHome task in InternalDistributionBwcSetupPlugin copies only gradle.properties and init.d/ into that project-named directory — not the wrapper/ subdirectory — so the inner build has no cached Gradle distribution and falls back to downloading from services.gradle.org. When that endpoint is temporarily unreachable (HTTP 504), the test fails.

The ForbiddenApisPrecommitPluginFuncTest failure is a separate configuration-cache serialisation bug: CheckForbiddenApisTask.getJavaLauncher() (@Nested @Optional Property<JavaLauncher>) cannot be serialised by Gradle's configuration cache when a Java 22 toolchain launcher is set. Gradle fails at CC storage time and the test sees a build failure.

Release risk: low-risk

Both failures are test infrastructure bugs; no Elasticsearch production code is affected. The BWC wrapper download issue is a gap in CI test fixture setup code introduced with PR #152919 (Jul 6). The forbidden-apis CC serialisation issue is in CheckForbiddenApisTask (build-tools-internal), not in Elasticsearch runtime. All 5 tests are already muted. The risk level would only need to change if the CC serialisation bug were to manifest in end-users' local builds — but CheckForbiddenApisTask is a CI precommit check, not a user-facing API.

Impact

• Affected branches: main
• Affected OS: Linux (ubuntu-2404 CI agents)
• Affected pipelines: elasticsearch-intake (build-logic step), elasticsearch-pull-request (build-logic step)
• Trend: Intermittent (12.5% fail rate in 16 executions per dashboard); all 5 tests appeared in the same build run (intake Associate sub-requests to their parent task in multi search API #44492 on 2026-07-24), consistent with a single transient services.gradle.org 504 that affected all 4 BWC tests in the same run

Reproduction

• Command: ./gradlew --no-daemon :build-tools-internal:integTest --tests "org.elasticsearch.gradle.internal.InternalDistributionBwcSetupPluginFuncTest" -Dtests.mutes.enabled=false
• Commit: 1ea5a7bb4521857f1ae75ebb596b737bdd3ec746
• Remote job: https://buildkite.com/elastic/elasticsearch-agentic-workflow/builds/188#019f94fa-0d42-4d6f-b75e-00f20584624e — ubuntu-2404
• Result: Not reproduced (tests passed — expected: failure only occurs when services.gradle.org returns 504, which is a transient condition; tests are also muted in the current codebase)
• Build scan: https://gradle-enterprise.elastic.co/s/lgfrrmcrlriic

Mitigation Proposal

Fix 1 — InternalDistributionBwcSetupPluginFuncTest (4 tests, wrapper download failure):

The setupGradleUserHome task in InternalDistributionBwcSetupPlugin.java (lines 219–232) must also copy the Gradle wrapper cache:

copy.from(absoluteGradleUserHomePath, copySpec -> {
    copySpec.include("gradle.properties");
    copySpec.include("init.d/*");
    copySpec.include("wrapper/**");   // ← add this line
});

This ensures the inner BWC ./gradlew -g gradleUserHome-major1 finds the pre-cached Gradle distribution and never calls services.gradle.org during tests.

Fix 2 — ForbiddenApisPrecommitPluginFuncTest (CC serialisation failure):

CheckForbiddenApisTask.getJavaLauncher() is annotated @Nested @Optional on a Property<JavaLauncher>. Gradle's configuration cache cannot serialise DefaultProperty<JavaLauncher>. Options:

1. Switch the property type to hold only the launcher's executable path as @Input Property<File> or @InputFile RegularFileProperty — which is CC-serialisable.
2. Mark it @Internal and ensure the task is excluded from the configuration cache, or suppress CC for the specific task.

• Reproduction verdict: Reproduction job passed (no 504 encountered on this run), consistent with the intermittent nature of the failure. Root cause is confirmed from the test report XML in the original CI artifact: "Server returned HTTP response code: 504 for URL: https://services.gradle.org/distributions/gradle-9.6.1-bin.zip" (BWC tests) and "error writing value of type 'org.gradle.api.internal.provider.DefaultProperty'" (ForbiddenApis test).
• Codebase investigation: InternalDistributionBwcSetupPlugin.java lines 219–232 (setupGradleUserHome missing wrapper/** include), BwcSetupExtension.java line 135 (-g gradleUserHome-<projectName>), AbstractGitAwareGradleFuncTest.groovy (copies wrapper only to gradleUserHome/), CheckForbiddenApisTask.java lines 303–308 (@Nested @Optional Property<JavaLauncher>), PR [Gradle] Simplify build tool test setup #152919 (Jul 6, introduced the gap), PR [Gradle] Run CheckForbiddenApisTask in a toolchain-matched forked process #152133+Add build support for using MemorySegment without --enable-preview on JDK 21 #151726 (Jun 25–26, introduced the CC issue).
• Proposed fix: Two separate small changes as described above.
• Effort estimate: small — both fixes are localised changes (1–5 lines each).
• Confidence: 0.95 (high) — root cause confirmed directly from the CI test report XML artifact.