Pin lang-SDK k8s test's Go/Java SDKs to upstream main#69545
Merged
jason810496 merged 3 commits intoJul 15, 2026
Conversation
boring-cyborg
Bot
added
area:dev-tools
area:kubernetes-tests
backport-to-v3-3-test
jason810496
force-pushed
the
ci/k8s-test/improve-lang-sdk-test
branch
from
July 9, 2026 05:24
84baf48 to
508ce82
Compare
jason810496
marked this pull request as ready for review
July 9, 2026 05:24
jason810496
requested review from
amoghrajesh,
ashb,
bugraoz93,
choo121600,
ephraimbuddy,
gopidesupavan,
jedcunningham,
jscheffl,
potiuk and
vatsrahul1001
as code owners
July 9, 2026 05:24
jason810496
marked this pull request as draft
July 9, 2026 09:54
jason810496
marked this pull request as ready for review
July 9, 2026 11:26
phanikumv
reviewed
Jul 10, 2026
Release/backport branches often lack go-sdk/java-sdk entirely or carry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would otherwise run the lang-SDK coordinator test against a missing or outdated SDK snapshot. The Go bundle and Java jar builds now always fetch go-sdk/java-sdk from upstream main regardless of the checked out branch, while airflow-core/task-sdk and the test's own harness fixtures keep tracking whatever branch is checked out. Fix lang-SDK Java build missing task-sdk sibling in native mode java-sdk's sdk/build.gradle.kts reads a sibling ../task-sdk/.../schema.json at build time. The upstream-main extraction only contains go-sdk/java-sdk, so building from it directly (native/CI toolchain mode) failed looking for a task-sdk directory that doesn't exist. Symlink the real, local task-sdk alongside the extraction so the reference resolves, without pulling task-sdk itself from upstream. Restore lang-SDK k8s test's gradle wrapper jar dropped by export-ignore java-sdk/.gitattributes marks gradle/wrapper/gradle-wrapper.jar export-ignore so ASF source-release tarballs stay free of compiled binaries (LEGAL-570). git archive, used to pull go-sdk/java-sdk out of upstream main for this test, respects that attribute and silently drops the jar, leaving the extracted java-sdk's ./gradlew unable to find itself and failing every build. Shorten gradle-wrapper.jar restoration comments
…8s test Upstream main's java-sdk/.gitattributes export-ignores gradlew and gradlew.bat (ASF LEGAL-570) in addition to the wrapper jar, so the git-archive extraction of upstream sources no longer contains the wrapper script the Java build invokes, failing CI with FileNotFoundError: './gradlew'. Restore all three wrapper files from the pinned commit instead of only the jar.
In dry-run mode run_command skips execution and returns stdout="" (a str), so every filesystem step consuming a command's output crashed: write_bytes on the gradle wrapper restore raised TypeError, the Go bundle copytree/copy raised FileNotFoundError, the Java jar glob hit sys.exit(1), and the artifact upload raised StopIteration. Guard those steps so --dry-run prints the full provisioning command sequence and completes cleanly.
jason810496
force-pushed
the
ci/k8s-test/improve-lang-sdk-test
branch
from
July 13, 2026 06:45
4526464 to
5b489ec
Compare
phanikumv
approved these changes
Jul 14, 2026
Contributor
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
github-actions Bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 15, 2026
…ache#69545) * Pin lang-SDK k8s test's Go/Java SDKs to upstream main Release/backport branches often lack go-sdk/java-sdk entirely or carry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would otherwise run the lang-SDK coordinator test against a missing or outdated SDK snapshot. The Go bundle and Java jar builds now always fetch go-sdk/java-sdk from upstream main regardless of the checked out branch, while airflow-core/task-sdk and the test's own harness fixtures keep tracking whatever branch is checked out. Fix lang-SDK Java build missing task-sdk sibling in native mode java-sdk's sdk/build.gradle.kts reads a sibling ../task-sdk/.../schema.json at build time. The upstream-main extraction only contains go-sdk/java-sdk, so building from it directly (native/CI toolchain mode) failed looking for a task-sdk directory that doesn't exist. Symlink the real, local task-sdk alongside the extraction so the reference resolves, without pulling task-sdk itself from upstream. Restore lang-SDK k8s test's gradle wrapper jar dropped by export-ignore java-sdk/.gitattributes marks gradle/wrapper/gradle-wrapper.jar export-ignore so ASF source-release tarballs stay free of compiled binaries (LEGAL-570). git archive, used to pull go-sdk/java-sdk out of upstream main for this test, respects that attribute and silently drops the jar, leaving the extracted java-sdk's ./gradlew unable to find itself and failing every build. Shorten gradle-wrapper.jar restoration comments * Restore gradle wrapper scripts dropped by export-ignore in lang-SDK k8s test Upstream main's java-sdk/.gitattributes export-ignores gradlew and gradlew.bat (ASF LEGAL-570) in addition to the wrapper jar, so the git-archive extraction of upstream sources no longer contains the wrapper script the Java build invokes, failing CI with FileNotFoundError: './gradlew'. Restore all three wrapper files from the pinned commit instead of only the jar. * Fix breeze k8s setup-lang-sdk-test crash with --dry-run In dry-run mode run_command skips execution and returns stdout="" (a str), so every filesystem step consuming a command's output crashed: write_bytes on the gradle wrapper restore raised TypeError, the Go bundle copytree/copy raised FileNotFoundError, the Java jar glob hit sys.exit(1), and the artifact upload raised StopIteration. Guard those steps so --dry-run prints the full provisioning command sequence and completes cleanly. (cherry picked from commit c22cc6d) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 15, 2026
…ache#69545) * Pin lang-SDK k8s test's Go/Java SDKs to upstream main Release/backport branches often lack go-sdk/java-sdk entirely or carry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would otherwise run the lang-SDK coordinator test against a missing or outdated SDK snapshot. The Go bundle and Java jar builds now always fetch go-sdk/java-sdk from upstream main regardless of the checked out branch, while airflow-core/task-sdk and the test's own harness fixtures keep tracking whatever branch is checked out. Fix lang-SDK Java build missing task-sdk sibling in native mode java-sdk's sdk/build.gradle.kts reads a sibling ../task-sdk/.../schema.json at build time. The upstream-main extraction only contains go-sdk/java-sdk, so building from it directly (native/CI toolchain mode) failed looking for a task-sdk directory that doesn't exist. Symlink the real, local task-sdk alongside the extraction so the reference resolves, without pulling task-sdk itself from upstream. Restore lang-SDK k8s test's gradle wrapper jar dropped by export-ignore java-sdk/.gitattributes marks gradle/wrapper/gradle-wrapper.jar export-ignore so ASF source-release tarballs stay free of compiled binaries (LEGAL-570). git archive, used to pull go-sdk/java-sdk out of upstream main for this test, respects that attribute and silently drops the jar, leaving the extracted java-sdk's ./gradlew unable to find itself and failing every build. Shorten gradle-wrapper.jar restoration comments * Restore gradle wrapper scripts dropped by export-ignore in lang-SDK k8s test Upstream main's java-sdk/.gitattributes export-ignores gradlew and gradlew.bat (ASF LEGAL-570) in addition to the wrapper jar, so the git-archive extraction of upstream sources no longer contains the wrapper script the Java build invokes, failing CI with FileNotFoundError: './gradlew'. Restore all three wrapper files from the pinned commit instead of only the jar. * Fix breeze k8s setup-lang-sdk-test crash with --dry-run In dry-run mode run_command skips execution and returns stdout="" (a str), so every filesystem step consuming a command's output crashed: write_bytes on the gradle wrapper restore raised TypeError, the Go bundle copytree/copy raised FileNotFoundError, the Java jar glob hit sys.exit(1), and the artifact upload raised StopIteration. Guard those steps so --dry-run prints the full provisioning command sequence and completes cleanly. (cherry picked from commit c22cc6d) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
joshuabvarghese
pushed a commit
to joshuabvarghese/airflow
that referenced
this pull request
Jul 16, 2026
* Pin lang-SDK k8s test's Go/Java SDKs to upstream main Release/backport branches often lack go-sdk/java-sdk entirely or carry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would otherwise run the lang-SDK coordinator test against a missing or outdated SDK snapshot. The Go bundle and Java jar builds now always fetch go-sdk/java-sdk from upstream main regardless of the checked out branch, while airflow-core/task-sdk and the test's own harness fixtures keep tracking whatever branch is checked out. Fix lang-SDK Java build missing task-sdk sibling in native mode java-sdk's sdk/build.gradle.kts reads a sibling ../task-sdk/.../schema.json at build time. The upstream-main extraction only contains go-sdk/java-sdk, so building from it directly (native/CI toolchain mode) failed looking for a task-sdk directory that doesn't exist. Symlink the real, local task-sdk alongside the extraction so the reference resolves, without pulling task-sdk itself from upstream. Restore lang-SDK k8s test's gradle wrapper jar dropped by export-ignore java-sdk/.gitattributes marks gradle/wrapper/gradle-wrapper.jar export-ignore so ASF source-release tarballs stay free of compiled binaries (LEGAL-570). git archive, used to pull go-sdk/java-sdk out of upstream main for this test, respects that attribute and silently drops the jar, leaving the extracted java-sdk's ./gradlew unable to find itself and failing every build. Shorten gradle-wrapper.jar restoration comments * Restore gradle wrapper scripts dropped by export-ignore in lang-SDK k8s test Upstream main's java-sdk/.gitattributes export-ignores gradlew and gradlew.bat (ASF LEGAL-570) in addition to the wrapper jar, so the git-archive extraction of upstream sources no longer contains the wrapper script the Java build invokes, failing CI with FileNotFoundError: './gradlew'. Restore all three wrapper files from the pinned commit instead of only the jar. * Fix breeze k8s setup-lang-sdk-test crash with --dry-run In dry-run mode run_command skips execution and returns stdout="" (a str), so every filesystem step consuming a command's output crashed: write_bytes on the gradle wrapper restore raised TypeError, the Go bundle copytree/copy raised FileNotFoundError, the Java jar glob hit sys.exit(1), and the artifact upload raised StopIteration. Guard those steps so --dry-run prints the full provisioning command sequence and completes cleanly.
potiuk
pushed a commit
that referenced
this pull request
Jul 19, 2026
…9545) * Pin lang-SDK k8s test's Go/Java SDKs to upstream main Release/backport branches often lack go-sdk/java-sdk entirely or carry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would otherwise run the lang-SDK coordinator test against a missing or outdated SDK snapshot. The Go bundle and Java jar builds now always fetch go-sdk/java-sdk from upstream main regardless of the checked out branch, while airflow-core/task-sdk and the test's own harness fixtures keep tracking whatever branch is checked out. Fix lang-SDK Java build missing task-sdk sibling in native mode java-sdk's sdk/build.gradle.kts reads a sibling ../task-sdk/.../schema.json at build time. The upstream-main extraction only contains go-sdk/java-sdk, so building from it directly (native/CI toolchain mode) failed looking for a task-sdk directory that doesn't exist. Symlink the real, local task-sdk alongside the extraction so the reference resolves, without pulling task-sdk itself from upstream. Restore lang-SDK k8s test's gradle wrapper jar dropped by export-ignore java-sdk/.gitattributes marks gradle/wrapper/gradle-wrapper.jar export-ignore so ASF source-release tarballs stay free of compiled binaries (LEGAL-570). git archive, used to pull go-sdk/java-sdk out of upstream main for this test, respects that attribute and silently drops the jar, leaving the extracted java-sdk's ./gradlew unable to find itself and failing every build. Shorten gradle-wrapper.jar restoration comments * Restore gradle wrapper scripts dropped by export-ignore in lang-SDK k8s test Upstream main's java-sdk/.gitattributes export-ignores gradlew and gradlew.bat (ASF LEGAL-570) in addition to the wrapper jar, so the git-archive extraction of upstream sources no longer contains the wrapper script the Java build invokes, failing CI with FileNotFoundError: './gradlew'. Restore all three wrapper files from the pinned commit instead of only the jar. * Fix breeze k8s setup-lang-sdk-test crash with --dry-run In dry-run mode run_command skips execution and returns stdout="" (a str), so every filesystem step consuming a command's output crashed: write_bytes on the gradle wrapper restore raised TypeError, the Go bundle copytree/copy raised FileNotFoundError, the Java jar glob hit sys.exit(1), and the artifact upload raised StopIteration. Guard those steps so --dry-run prints the full provisioning command sequence and completes cleanly. (cherry picked from commit c22cc6d) Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Release/backport branches (
v3-1-test,v3-2-test, ...) often lackgo-sdk/java-sdkentirely orcarry a stale, branch-cut-frozen copy, so backporting a core/task-sdk fix to such a branch would
silently run the lang-SDK coordinator k8s test against missing or outdated SDK sources.
What
_lang_sdk_fetch_upstream_sdk_sources(): shallow-fetches upstreammain(upstreamremote ifconfigured, else the canonical GitHub URL) and extracts
go-sdk/java-sdkinto a throwaway stagingdir via
git archive, never touching the real working tree.go_example's relative layout withgo-sdkswapped for the upstream copy (its
go.modreplaceis a filesystem path, so this avoids editingit). Java build simply points gradle at the upstream
java-sdkcopy instead of the local one.airflow-core/,task-sdk/, and the test's own harness fixtures (go_example/java_example) areuntouched and keep tracking the checked-out branch.
kubernetes_commands.py), so it applies to both CI and localbreeze k8s setup-lang-sdk-test— no CI workflow changes needed.Note
didn't before.
providers/will be handled in the follow-up (we should make the k8s test prod image reference the latest k8s provider IMO).Was generative AI tooling used to co-author this PR?