Docs: Document adaptive split sizing configurations#16557
Open
pratham76 wants to merge 1 commit into
Open
Conversation
Author
|
Local Rendering: |
Author
|
@RussellSpitzer @karuppayya Could you PTAL? Thanks! |
karuppayya
approved these changes
May 26, 2026
Author
|
@kevinjqliu @huaxingao Could you take a look. Thanks! |
| | spark.sql.iceberg.merge-schema | false | Enables modifying the table schema to match the write schema. Only adds columns missing columns | | ||
| | spark.sql.iceberg.report-column-stats | true | Report Puffin Table Statistics if available to Spark's Cost Based Optimizer. CBO must be enabled for this to be effective | | ||
| | spark.sql.iceberg.read.adaptive-split-size.enabled | Table default | Enables adaptive split sizing for read operations. When enabled, split size is automatically adjusted based on scan size and parallelism | | ||
| | spark.sql.iceberg.read.adaptive-split-size.parallelism | max(spark.default.parallelism, spark.sql.shuffle.partitions) | Overrides the parallelism used for adaptive split sizing. Must be greater than 0 | |
Member
There was a problem hiding this comment.
Rather than documenting spark config, I would just say "Spark's default parallelism"
Author
There was a problem hiding this comment.
Thanks @RussellSpitzer , have modified.
Updated Local rendering.
Author
There was a problem hiding this comment.
On another thought, the default value does not exactly correspond to spark's default parallelism in this case, as it is maximum of both values spark.default.parallelism and spark.sql.shuffle.partitions, thought of documenting it explicitly. Please do provide your thoughts on this. Thanks!
RussellSpitzer
approved these changes
May 26, 2026
pratham76
force-pushed
the
issue-16556
branch
3 times, most recently
from
7aeaca5 to
61b4540
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documenting the adaptive split sizing configurations that were added in #16088 but were missed to be documented.
Closes #16556