Skip to content

chore: bump Spark 3.5 to 3.5.9 - #5181

Merged
mbutrovich merged 1 commit into
apache:mainfrom
andygrove:bump-spark-3.5.9
Jul 31, 2026
Merged

chore: bump Spark 3.5 to 3.5.9#5181
mbutrovich merged 1 commit into
apache:mainfrom
andygrove:bump-spark-3.5.9

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A — routine patch-version bump within the already-supported Spark 3.5 line.

Rationale for this change

Spark 3.5 support should track the latest Spark 3.5 patch release. Apache Spark
3.5.9 has been released (tag v3.5.9), so this moves Comet's Spark 3.5 build,
CI, and documentation from 3.5.8 to 3.5.9.

What changes are included in this PR?

  • Bumps <spark.version> in the spark-3.5 Maven profile to 3.5.9. Spark 3.5.9
    makes no transitive dependency changes relative to 3.5.8 (scala.version,
    parquet.version, slf4j.version, jetty.version, hadoop.version and
    arrow.version are all unchanged), so no other profile properties move.
  • Renames dev/diffs/3.5.8.diff to dev/diffs/3.5.9.diff and regenerates it
    against the v3.5.9 tag. Three hunks needed manual reapplication:
    • pom.xml — context drift only (ivy.version bumped upstream).
    • WriteDistributionAndOrderingSuite.scala — context drift only (upstream
      added TransformExpression to a neighbouring import).
    • KeyGroupedPartitioningSuite.scala — context drift plus one real change:
      3.5.9 adds a new collectAllShuffles helper matching on
      ShuffleExchangeExec. Under Comet the shuffle is a
      CometShuffleExchangeExec, so the new helper's assert(allShuffles.nonEmpty)
      call sites would fail. It now matches ShuffleExchangeLike, consistent with
      how the existing collectShuffles helper is already patched.
  • Points the Spark SQL, Spark SQL writer, and Iceberg 1.9/1.10 CI jobs at 3.5.9,
    and updates dev/ci/compute-changes.py's spark_3_5 path filter to the new
    diff filename.
  • Updates the user guide's supported-version table, the Spark version
    compatibility page, the Iceberg contributor guide, the setup-spark-builder
    action / reusable-workflow input descriptions, and the pip install pyspark==
    examples in the PyArrow UDF test scripts.

How are these changes tested?

  • git apply --check dev/diffs/3.5.9.diff against a pristine v3.5.9 checkout
    passes with no rejects or fuzz.
  • Verified the regenerated diff touches exactly the same 73 files as the 3.5.8
    diff, and that its only content differences from 3.5.8 are the context lines
    above plus the collectAllShuffles change.
  • npx prettier --check on the modified markdown.
  • Full verification is the Spark SQL CI job on this PR, which clones v3.5.9,
    applies the new diff, and runs the SQL test suites.

Rename dev/diffs/3.5.8.diff to 3.5.9.diff and regenerate it against the
v3.5.9 tag, and point the build, CI workflows, and docs at 3.5.9.
@andygrove andygrove added this to the 1.0.0 milestone Jul 31, 2026

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI, thanks @andygrove!

@mbutrovich
mbutrovich merged commit 0abe966 into apache:main Jul 31, 2026
149 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants