Skip to content

chore: bump Spark 4.0 to 4.0.4 - #5182

Open
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:bump-spark-4.0.4
Open

chore: bump Spark 4.0 to 4.0.4#5182
andygrove wants to merge 2 commits into
apache:mainfrom
andygrove:bump-spark-4.0.4

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A — routine patch-version bump within the already-supported Spark 4.0 line.

Rationale for this change

Spark 4.0 support should track the latest Spark 4.0 patch release. Apache Spark
4.0.4 has been released (tag v4.0.4), so this moves Comet's Spark 4.0 build,
CI, and documentation from 4.0.2 to 4.0.4 (skipping 4.0.3).

What changes are included in this PR?

  • Bumps <spark.version> in the spark-4.0 Maven profile to 4.0.4. Spark 4.0.4
    makes no transitive dependency changes relative to 4.0.2 (scala.version,
    parquet.version, slf4j.version, jetty.version, hadoop.version and
    arrow.version are all unchanged), so no other profile properties move.
  • Renames dev/diffs/4.0.2.diff to dev/diffs/4.0.4.diff and regenerates it
    against the v4.0.4 tag. Three hunks needed manual reapplication, all pure
    import-line context drift with no behavioural change:
    • DataFrameWindowFunctionsSuite.scala — upstream added a
      catalyst.trees.UnaryLike import between the patched neighbours.
    • KeyGroupedPartitioningSuite.scala — upstream added RDDScanExec to a
      neighbouring import.
    • WriteDistributionAndOrderingSuite.scala — upstream added
      TransformExpression to a neighbouring import.
  • Updates the two version references inside the diff's HiveUDFDynamicLoadSuite
    comment. hive-test-udfs.jar is still stripped from the 4.0.4 release source
    tag (only a README remains under sql/hive/src/test/noclasspath/), so the
    IgnoreCometSuite mixin is still required.
  • Points the Spark SQL CI job and the PyArrow UDF job's pyspark pin at 4.0.4,
    and updates dev/ci/compute-changes.py's spark_4_0 path filter to the new
    diff filename.
  • Updates the user guide's supported-version table and the Spark version
    compatibility page.

How are these changes tested?

  • git apply --check dev/diffs/4.0.4.diff against a pristine v4.0.4 checkout
    passes with no rejects or fuzz.
  • Verified the regenerated diff touches exactly the same 88 files as the 4.0.2
    diff, and that its only content differences from 4.0.2 are the context lines
    and comment version strings above.
  • npx prettier --check on the modified markdown.
  • Full verification is the Spark SQL CI job, which is label-gated for Spark 4.0
    (run-spark-4.0-tests) — that label should be applied before merge.

Rename dev/diffs/4.0.2.diff to 4.0.4.diff and regenerate it against the
v4.0.4 tag, and point the build, CI workflows, and docs at 4.0.4.
@andygrove andygrove added this to the 1.0.0 milestone Jul 31, 2026

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI, thanks @andygrove!

# Conflicts:
#	docs/source/user-guide/latest/installation.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants