Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ jobs:
uses: ./.github/workflows/spark_sql_test_reusable.yml
with:
spark-short: '4.1'
spark-full: '4.1.2'
spark-full: '4.1.3'
java: 17

iceberg_1_8:
Expand Down Expand Up @@ -332,5 +332,5 @@ jobs:
iceberg-short: '1.11'
iceberg-full: '1.11.0'
spark-short: '4.1'
spark-full: '4.1.2'
spark-full: '4.1.3'
java: 17
2 changes: 1 addition & 1 deletion .github/workflows/pyarrow_udf_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ jobs:
pyspark: "4.0.2"
- name: Spark 4.1
maven_profiles: "-Pspark-4.1"
pyspark: "4.1.2"
pyspark: "4.1.3"
container:
# Pinned to the Debian 12 (bookworm) base so the system `python3` is 3.11. The default
# `amd64/rust` image is Debian 13 (trixie) which ships Python 3.13 and no python3.11 apt
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/spark_sql_writer_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ jobs:
# versions only; 3.4 and 4.0 are label-gated and not offered here).
case "${{ inputs.spark-version }}" in
3.5) spark_full=3.5.9; java=17 ;;
4.1) spark_full=4.1.2; java=17 ;;
4.1) spark_full=4.1.3; java=17 ;;
*) echo "Unsupported spark-version: ${{ inputs.spark-version }}" >&2; exit 1 ;;
esac
echo "spark-full=$spark_full" >> "$GITHUB_OUTPUT"
Expand Down
2 changes: 1 addition & 1 deletion dev/ci/compute-changes.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@
"!spark/src/main/spark-4.2/**",
"!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
"spark/pom.xml",
"dev/diffs/4.1.2.diff",
"dev/diffs/4.1.3.diff",
"pom.xml",
"rust-toolchain.toml",
".github/workflows/ci.yml",
Expand Down
78 changes: 39 additions & 39 deletions dev/diffs/4.1.2.diff → dev/diffs/4.1.3.diff

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions docs/source/contributor-guide/benchmarking_macos.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,13 @@ export DF_BENCH=`pwd`

## Install Spark

Install Apache Spark. This example refers to 4.1.2 version.
Install Apache Spark. This example refers to 4.1.3 version.

```shell
wget https://archive.apache.org/dist/spark/spark-4.1.2/spark-4.1.2-bin-hadoop3.tgz
tar xzf spark-4.1.2-bin-hadoop3.tgz
sudo mv spark-4.1.2-bin-hadoop3 /opt
export SPARK_HOME=/opt/spark-4.1.2-bin-hadoop3/
wget https://archive.apache.org/dist/spark/spark-4.1.3/spark-4.1.3-bin-hadoop3.tgz
tar xzf spark-4.1.3-bin-hadoop3.tgz
sudo mv spark-4.1.3-bin-hadoop3 /opt
export SPARK_HOME=/opt/spark-4.1.3-bin-hadoop3/
```

Start Spark in standalone mode:
Expand Down
2 changes: 1 addition & 1 deletion docs/source/contributor-guide/iceberg-spark-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ diff must be generated against its own tag.

The `iceberg_spark_test_<version>.yml` workflows apply these diffs and run the three Gradle targets above
against each Iceberg version. Iceberg 1.8.1 runs against Spark 3.4.3 with Java 11; Iceberg 1.9.1 and 1.10.0
run against Spark 3.5.9 with Java 17; Iceberg 1.11.0 runs against Spark 4.1.2 with Java 17. Iceberg 1.11
run against Spark 3.5.9 with Java 17; Iceberg 1.11.0 runs against Spark 4.1.3 with Java 17. Iceberg 1.11
(the only version testing Spark 4.1) runs on every pull request and on pushes to main; the older versions
(1.8, 1.9, 1.10) run only on pushes to main, or on a pull request labeled `run-iceberg-tests`. All caller
workflows delegate to `iceberg_spark_test_reusable.yml`, which holds the build and test job logic.
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Spark 4.0.2 is supported with Java 17 and Scala 2.13.

## Spark 4.1

Spark 4.1.2 is supported with Java 17/21 and Scala 2.13.
Spark 4.1.3 is supported with Java 17/21 and Scala 2.13.

### Known Limitations

Expand Down
2 changes: 1 addition & 1 deletion docs/source/user-guide/latest/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ We recommend moving to JDK 17 or later and Spark 3.5 or later.
| 3.4.3 | 11/17 | 2.12/2.13 | Yes | Yes |
| 3.5.9 | 11/17 | 2.12/2.13 | Yes | Yes |
| 4.0.2 | 17/21 | 2.13 | Yes | Yes |
| 4.1.2 | 17/21 | 2.13 | Yes | Yes |
| 4.1.3 | 17/21 | 2.13 | Yes | Yes |

Note that we do not test the full matrix of supported Java and Scala versions in CI for every Spark version.

Expand Down
8 changes: 4 additions & 4 deletions docs/source/user-guide/latest/kubernetes.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ spec:
image: apache/datafusion-comet:$COMET_VERSION-spark3.5.5-scala2.12-java11
imagePullPolicy: IfNotPresent
mainClass: org.apache.spark.examples.SparkPi
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.13-4.1.2.jar
mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.13-4.1.3.jar
sparkConf:
"spark.executor.extraClassPath": "/opt/spark/jars/comet-spark-spark3.5_2.12-$COMET_VERSION.jar"
"spark.driver.extraClassPath": "/opt/spark/jars/comet-spark-spark3.5_2.12-$COMET_VERSION.jar"
Expand All @@ -82,17 +82,17 @@ spec:
"spark.comet.shuffle.enabled": "true"
"spark.comet.shuffle.mode": "auto"
"spark.shuffle.manager": "org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager"
sparkVersion: 4.1.2
sparkVersion: 4.1.3
driver:
labels:
version: 4.1.2
version: 4.1.3
cores: 1
coreLimit: 1200m
memory: 512m
serviceAccount: spark-operator-spark
executor:
labels:
version: 4.1.2
version: 4.1.3
instances: 1
cores: 1
coreLimit: 1200m
Expand Down
4 changes: 2 additions & 2 deletions kube/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# limitations under the License.
#

FROM apache/spark:4.1.2 AS builder
FROM apache/spark:4.1.3 AS builder

USER root

Expand Down Expand Up @@ -69,7 +69,7 @@ RUN mkdir -p /root/.m2 && \
RUN cd /comet \
&& JAVA_HOME=$(readlink -f $(which javac) | sed "s/\/bin\/javac//") make release-nogit PROFILES="-Pspark-$SPARK_VERSION -Pscala-$SCALA_VERSION"

FROM apache/spark:4.1.2
FROM apache/spark:4.1.3
ENV SPARK_VERSION=4.1
ENV SCALA_VERSION=2.13
USER root
Expand Down
4 changes: 2 additions & 2 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ under the License.
<scala.plugin.version>4.9.6</scala.plugin.version>
<scalatest.version>3.2.16</scalatest.version>
<scalatest-maven-plugin.version>2.2.0</scalatest-maven-plugin.version>
<spark.version>4.1.2</spark.version>
<spark.version>4.1.3</spark.version>
<spark.version.short>4.1</spark.version.short>
<spark.maven.scope>provided</spark.maven.scope>
<protobuf.version>3.25.5</protobuf.version>
Expand Down Expand Up @@ -717,7 +717,7 @@ under the License.
yet published, so the -Psemanticdb / scalafix lint job is skipped for spark-4.1. -->
<scala.version>2.13.17</scala.version>
<scala.binary.version>2.13</scala.binary.version>
<spark.version>4.1.2</spark.version>
<spark.version>4.1.3</spark.version>
<spark.version.short>4.1</spark.version.short>
<parquet.version>1.16.0</parquet.version>
<semanticdb.version>4.13.6</semanticdb.version>
Expand Down
2 changes: 1 addition & 1 deletion spark/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,7 @@ under the License.
<version>1.11.0</version>
<scope>test</scope>
</dependency>
<!-- Jetty 11.x for Spark 4.1 (jakarta.servlet); matches Spark 4.1.2's jetty.version -->
<!-- Jetty 11.x for Spark 4.1 (jakarta.servlet); matches Spark 4.1.3's jetty.version -->
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
Expand Down
Loading