Skip to content

chore: bump Spark 4.1 to 4.1.3 - #5183

Merged
andygrove merged 2 commits into
apache:mainfrom
andygrove:bump-spark-4.1.3
Aug 1, 2026
Merged

chore: bump Spark 4.1 to 4.1.3#5183
andygrove merged 2 commits into
apache:mainfrom
andygrove:bump-spark-4.1.3

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A — routine patch-version bump within the already-supported Spark 4.1 line.

Rationale for this change

Spark 4.1 support should track the latest Spark 4.1 patch release. Apache Spark
4.1.3 has been released (tag v4.1.3), so this moves Comet's Spark 4.1 build,
CI, Docker image, and documentation from 4.1.2 to 4.1.3. Spark 4.1 is also
Comet's default profile, so this bumps the top-level <spark.version> too.

What changes are included in this PR?

  • Bumps the default <spark.version> and the spark-4.1 profile's
    <spark.version> to 4.1.3. Spark 4.1.3 makes no transitive dependency changes
    relative to 4.1.2 (scala.version 2.13.17, parquet.version 1.16.0,
    slf4j.version 2.0.17, jetty.version 11.0.26, hadoop.version 3.4.2,
    arrow.version 18.3.0 are all unchanged), so no other profile properties move
    and spark/pom.xml's pinned Jetty 11.0.26 test deps still match — only the
    comment naming the reference version is updated.
  • Renames dev/diffs/4.1.2.diff to dev/diffs/4.1.3.diff and regenerates it
    against the v4.1.3 tag. Two hunks needed manual reapplication, both pure
    import-line context drift with no behavioural change:
    • DataFrameWindowFunctionsSuite.scala — upstream added a
      catalyst.trees.UnaryLike import between the patched neighbours.
    • StreamRealTimeModeAllowlistSuite.scala — upstream moved
      LowLatencyMemoryStream from execution.streaming to
      execution.streaming.sources.
  • Points the Spark SQL, Spark SQL writer, Iceberg 1.11, and PyArrow UDF CI jobs
    at 4.1.3, and updates dev/ci/compute-changes.py's spark_4_1 path filter to
    the new diff filename.
  • Bumps kube/Dockerfile to apache/spark:4.1.3 and updates the Kubernetes,
    macOS benchmarking, installation, compatibility, and Iceberg docs.

How are these changes tested?

  • git apply --check dev/diffs/4.1.3.diff against a pristine v4.1.3 checkout
    passes with no rejects or fuzz.
  • Verified the regenerated diff touches exactly the same 96 files as the 4.1.2
    diff, and that its only content differences from 4.1.2 are the two context
    lines above.
  • npx prettier --check on the modified markdown.
  • Full verification is the Spark SQL and Iceberg 1.11 CI jobs on this PR, which
    clone v4.1.3, apply the new diff, and run the test suites.

One thing to confirm on review

Maven Central, PyPI, and Docker Hub are all unreachable from the environment I
prepared this in, so I could not independently confirm that the
org.apache.spark:*:4.1.3 artifacts, pyspark==4.1.3, and the
apache/spark:4.1.3 image are all published yet — the v4.1.3 tag (identical to
v4.1.3-rc1, i.e. the vote passed on rc1) is the only evidence I could gather.
CI will fail fast on the Maven and PyPI pins if either is not yet available. The
apache/spark Docker image sometimes lags the release, and kube/Dockerfile is
not exercised by CI, so that tag is worth an explicit check before merge.

Rename dev/diffs/4.1.2.diff to 4.1.3.diff and regenerate it against the
v4.1.3 tag, and point the build, CI workflows, Docker image, and docs at
4.1.3.
@andygrove andygrove added this to the 1.0.0 milestone Jul 31, 2026

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI, thanks @andygrove!

Resolved conflicts in .github/workflows/spark_sql_writer_tests.yml and
docs/source/contributor-guide/iceberg-spark-tests.md by keeping main's
Spark 3.5.9 bump alongside this branch's Spark 4.1.3 bump.
@andygrove
andygrove merged commit 064dcf6 into apache:main Aug 1, 2026
72 checks passed
@andygrove
andygrove deleted the bump-spark-4.1.3 branch August 1, 2026 11:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants