Parquet reader rejects canonical UNKNOWN logical type on BOOLEAN physical columns

[cdelmonte-zg]

This does not appear to be a duplicate of #8776 / #8777. Those addressed LogicalType::_Unknown { field_id }, i.e. the forward-compatibility variant used when the reader encounters an unknown logical-type union value.

This issue is about LogicalType::Unknown, the canonical Parquet UNKNOWN logical type. In arrow-rs these are separate enum variants in parquet/src/basic.rs.

To Reproduce

The Delta Acceptance Tests contain expected-output Parquet files for void / NullType workloads. These files appear to contain columns encoded as BOOLEAN physical type annotated with LogicalType::Unknown.

Reading such a file with arrow-rs fails during schema validation:

let f = File::open("spark_void.parquet").unwrap();
let reader = ParquetRecordBatchReaderBuilder::try_new(f)?;

This returns:

General("Cannot annotate Unknown from BOOLEAN for field 'void_col'")

A fully programmatic in-memory repro is awkward because the type builder rejects this physical/logical type combination at write time. The issue is therefore only visible when reading a file produced by an external writer.

Example files are available in the https://github.com/delta-incubator/dat (DAT) repository. The workloads are shipped as the https://github.com/delta-incubator/dat/releases/download/v0.04-preview/v0.0.4_dat_workloads.tar.gz. After extracting, the relevant cases are void_001_void_top_level, void_002_void_nested_struct, void_005_void_schema_evolution, void_006_void_multiple_columns, and void_007_void_with_backticks. The Parquet files that trigger this issue live in each workload's delta/ subdirectory.

Expected behavior

My understanding is that LogicalType::Unknown represents a column whose values are always null. If so, the Parquet reader should probably accept this annotation independently of the physical type and expose the column as DataType::Null.

Please correct me if this interpretation of UNKNOWN is wrong.

Possible fix direction

A fix may need to:

1. Broaden the validator in parquet/src/schema/types.rs so that LogicalType::Unknown is not restricted to INT32.
2. Ensure parquet/src/arrow/schema/primitive.rs maps LogicalType::Unknown to DataType::Null for all primitive physical types. from_int32 already appears to handle this; the other primitive type mappers do not.

Versions

• parquet = "58.1.0"
• Also reproduced with parquet = "57.3.0"
• Also reproduced on current main

Component(s)

Parquet

Context

This blocks delta-io/delta-kernel-rs#1858 from running the Delta Acceptance Tests for void / NullType columns end-to-end.

The Delta tables themselves load fine because Delta does not materialize void columns in data Parquet files. However, the Delta Acceptance Tests harness reads the expected-output Parquet files written by Spark, and those files can contain BOOLEAN + UNKNOWN columns.

[etseidl]

Thanks for the detailed report. I've had issues with UNKNOWN on another project, where again there was an implicit assumption that the physical type had to be INT32 to use that annotation. But the actual spec seems to be silent on that. Do you have time to explore a fix? If not, I can take a look.

Nevermind, I got bored and gave it a stab.

[etseidl]

Example files are available in the Delta Acceptance Tests repository under void_NNN_* workloads, in expected_data/.

Is the DAT repo private? The link just gives me a 404 (but maybe that's general github brokenness?).

[cdelmonte-zg]

Example files are available in the Delta Acceptance Tests repository under void_NNN_* workloads, in expected_data/.

Is the DAT repo private? The link just gives me a 404 (but maybe that's general github brokenness?).

Apologies -- typo in the issue body. The repo is delta-incubator/dat (public), not delta-io/dat. I'll edit the link.
The void workloads are not checked into main. They're shipped as an asset of the v0.04-preview release tarball:
https://github.com/delta-incubator/dat/releases/download/v0.04-preview/v0.0.4_dat_workloads.tar.gz

After extracting, the cases are under void_001_void_top_level/, void_002_void_nested_struct/, void_005_void_schema_evolution/, void_006_void_multiple_columns/, and
void_007_void_with_backticks/. The Parquet files that trigger this issue live in each workload's delta/ subdirectory

[cdelmonte-zg]

Thanks a lot @etseidl , really appreciate you picking this up so quickly and putting #9855 together the same day!

Quick confirmation: I patched delta-kernel-rs locally to pull parquet and the arrow-* workspace crates from your logical_type_unknown branch, and all 5 void DAT _read_all workloads currently in the EXPECTED_KERNEL_FAILURES now pass:

• void_001_void_top_level_read_all
• void_002_void_nested_struct_read_all
• void_005_void_schema_evolution_read_all
• void_006_void_multiple_columns_read_all
• void_007_void_with_backticks_read_all

That covers top-level, nested-struct, schema-evolution, and column-mapping variants.

I think, this confirms that the fix unblocks delta-kernel-rs#1858 end-to-end.

[etseidl]

Thanks for the confirmation @cdelmonte-zg. I'll mark the PR ready for review.

[alamb]

label_issue.py automatically added labels {'parquet'} from #9855