WIP: Update to arrow/parquet 59.2.0 by alamb · Pull Request #24030 · apache/datafusion

alamb · 2026-07-31T15:19:52Z

Which issue does this PR close?

Related to Release arrow-rs / parquet Minor version 59.2.0 (July 2026) arrow-rs#9879

Rationale for this change

Upgrade to latest arrow / parquet library

What changes are included in this PR?

Update version pins
Update for API deprecation (I will comment inline)

Are these changes tested?

yes, by CI

Are there any user-facing changes?

Not yet

github-actions · 2026-07-31T15:49:30Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
error: running 'cargo update' on crate 'datafusion-common' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-common v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)`
    ... which satisfies path dependency `datafusion-common` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_common-54_1_0-x86_64_unknown_linux_gnu-a582417a7285e6f6)`

-----
error: failed to update dependencies for crate datafusion-common v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/common --features backtrace,force_hash_collisions,object_store,parquet,parquet_encryption,recursive_protection,sql,sqlparser &&
          cargo update

error: running 'cargo update' on crate 'datafusion-datasource-parquet' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-datasource-parquet v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)`
    ... which satisfies path dependency `datafusion-datasource-parquet` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_datasource_parquet-54_1_0-x86_64_unknown_linux_gnu-709159958f346fac)`

-----
error: failed to update dependencies for crate datafusion-datasource-parquet v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet --features parquet_encryption &&
          cargo update

error: running 'cargo update' on crate 'datafusion-functions' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-functions v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)`
    ... which satisfies path dependency `datafusion-functions` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_functions-54_1_0-x86_64_unknown_linux_gnu-15b8b9894b75340c)`

-----
error: failed to update dependencies for crate datafusion-functions v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/functions --features base64,blake2,blake3,chrono-tz,crypto_expressions,datetime_expressions,default,encoding_expressions,hex,math_expressions,md-5,regex,regex_expressions,sha2,string_expressions,unicode_expressions,uuid &&
          cargo update

error: running 'cargo update' on crate 'datafusion-functions-aggregate' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-functions-aggregate v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)`
    ... which satisfies path dependency `datafusion-functions-aggregate` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_functions_aggregate-54_1_0-x86_64_unknown_linux_gnu-12bb28bb2ce9d807)`

-----
error: failed to update dependencies for crate datafusion-functions-aggregate v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate --features force_hash_collisions &&
          cargo update

error: aborting due to failure to run 'cargo update' for crate datafusion-common v54.1.0

alamb · 2026-07-31T15:39:18Z

-        if let Some(hint) = metadata_size_hint {
-            inner = inner.with_footer_size_hint(hint)
-        };
+        let location = partitioned_file.object_meta.location;


ParquetObjectReader is deprecated due to

Remove object_store dependency - Extract object store related code arrow-rs#10308

alamb · 2026-07-31T15:40:26Z

    filename: String,
    metadata: Arc<ParquetMetaData>,
-    inner: ParquetObjectReader,
+    object_store: Arc<dyn ObjectStore>,


this is arguable a nicer example now, as it avoids having to wrap the ParquetObjectReader

arrow-rs formalized MapBuilder's default field names from keys/values to key/value to match the Arrow spec (apache/arrow-rs#10297). Update the hardcoded schema in arrow_field()/arrow_metadata() and the nested_struct cast test to match, fixing a StructArray::new panic on mismatched field names.

alamb · 2026-07-31T20:30:24Z

        let to_write = RecordBatch::try_from_iter(data).unwrap();
        let path = object_store::path::Path::from(format!("{file_num}.parquet").as_str());
-        let object_store_writer = ParquetObjectWriter::new(Arc::clone(&store) as _, path);
+        let object_store_writer = BufWriter::new(Arc::clone(&store) as _, path);


ParquetObjectWriter wsas deprecated: apache/arrow-rs#10308

ParquetObjectWriter is a thin wrapper around BufWriter: https://docs.rs/parquet/latest/src/parquet/arrow/async_writer/store.rs.html#72-74

alamb · 2026-07-31T20:32:08Z

    ) -> BoxFuture<'_, datafusion::parquet::errors::Result<Bytes>> {
        println!("get_bytes: {} Reading range {:?}", self.filename, range);
-        self.inner.get_bytes(range)
+        let object_store = Arc::clone(&self.object_store);


This is due to inlining the logic for ParquetObjectReader

It turns out that DataFusion already has its own copy of the logic of the ParquetObjectReader between ParquetFileReader and CachedParquetFileReader. I have a PR that will unify them:

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

alamb · 2026-07-31T20:32:58Z

                            struct_type(vec![
-                                non_null_field("keys", DataType::Utf8),
-                                field("values", DataType::Int32),
+                                non_null_field("key", DataType::Utf8),


This change is needed due to

chore: formalize the default map field names to match default arrow spec arrow-rs#10297
I think we should probably revert that for this minor release and I am working on that upstream

alamb · 2026-07-31T20:33:44Z

        options: Option<&'a ArrowReaderOptions>,
    ) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {
-        self.inner.get_metadata(options)
+        let object_meta = self.partitioned_file.object_meta.clone();


THe need for this replication is removed by unifying the structures, as I propose to do do in

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

alamb · 2026-07-31T20:34:17Z

+
+**Migration guide:**
+
+If your [`AsyncFileReader`] implementation exists mainly to read from an


This will need to get updated if we merge this one first

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

…rrow-rs 59.2.0 Bumping arrow-rs changed the underlying buffer memory footprint reported by ArrayData::get_slice_memory_size() for this scenario, shifting the accumulator's accounted size from 282 to 290 bytes. This is an expected size increase from the dependency bump, not an accounting regression: the merge behavior and both sibling tests (does_not_over_account_memory_distinct/_ordered) are unaffected.

alamb · 2026-07-31T21:01:12Z

        acc1 = merge(acc1, acc2)?;

-        assert_eq!(acc1.size(), 282);
+        assert_eq!(acc1.size(), 290);


Due to

fix off by one error for slice accounting arrow-rs#10406

alamb added 2 commits July 31, 2026 11:16

TEMP: Update to arrow/parquet 59.2.0

63e2a79

chore: Update Cargo more

74895bf

alamb mentioned this pull request Jul 31, 2026

Release arrow-rs / parquet Minor version 59.2.0 (July 2026) apache/arrow-rs#9879

Open

9 tasks

chore: API

b23a761

github-actions Bot added the datasource Changes to the datasource crate label Jul 31, 2026

comments about metadata hint

d81cf4d

github-actions Bot added the auto detected api change Auto detected API change label Jul 31, 2026

Better refactor

e575b6a

github-actions Bot added the documentation Improvements or additions to documentation label Jul 31, 2026

missing link

bda399c

alamb mentioned this pull request Jul 31, 2026

chore: formalize the default map field names to match default arrow spec apache/arrow-rs#10297

Merged

alamb commented Jul 31, 2026

View reviewed changes

alamb mentioned this pull request Jul 31, 2026

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

Open

github-actions Bot added common Related to common crate functions Changes to functions implementation labels Jul 31, 2026

alamb commented Jul 31, 2026

View reviewed changes

alamb mentioned this pull request Jul 31, 2026

Revert "chore: formalize the default map field names to match default arrow spec (#10297)" apache/arrow-rs#10506

Open

alamb commented Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Update to arrow/parquet 59.2.0 - #24030

WIP: Update to arrow/parquet 59.2.0#24030
alamb wants to merge 8 commits into
apache:mainfrom
alamb:alamb/update_arrow_59.2.0

alamb commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant


		Migration guide:

		If your [`AsyncFileReader`] implementation exists mainly to read from an

Uh oh!

Conversation

alamb commented Jul 31, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jul 31, 2026 •

edited

Loading

alamb Jul 31, 2026 •

edited

Loading