[Parquet] Restore pre-#8715 tolerance for mixed row-group `ordinal` metadata (or add a lenient reader flag)

[zhuqi-lucas]

cc @alamb

Which issue does this help address?

PR #8715 (merged 2025-11-14) shipped a new OrdinalAssigner::ensure invariant that rejects Parquet files whose row-groups disagree on whether ordinal is populated. Before #8715, the reader silently tolerated mixed row-group ordinal metadata (either uniform ordinals, no ordinals, or a mix), assigning sequential ordinals when needed.

The ordinal field is defined as optional on RowGroup in the parquet-format Thrift spec, and there is no clause requiring it to be uniformly present or absent across a file. Any file that was valid pre-#8715 is still valid per spec today.

Real-world impact

Our product hit this on production reference-data parquet produced by a Go-based writer (Go parquet libraries commonly emit mixed-ordinal RowGroup metadata as they flush row-groups incrementally). The exact error:

Parquet error: Inconsistent ordinal assignment: first_has_ordinal is set to
false but row-group with actual ordinal 1 has rg_has_ordinal set to true

Files that read fine on parquet 57.0.0 stopped reading once we bumped to 57.1+ / 58.x. There is no downgrade path that keeps DataFusion 54/55 available.

This is likely to bite other users too — cross-language Parquet ecosystems (Go parquet libs, older Java, custom writers) produce mixed-ordinal files in the wild.

Why #8715 tightened this

#8715 added row-number computation as a virtual column (for Delta Lake / Iceberg deletion vectors, per issue #7299). Correct row-numbering requires knowing each row-group's position in the file, so the PR added strict validation that ordinals are consistent.

That reasoning holds only when a caller actually uses the row-number virtual column. For the vast majority of readers that never touch that feature, the pre-#8715 tolerant behavior was correct and useful.

Proposal

Add a reader option (e.g. ArrowReaderOptions::with_lenient_row_group_ordinal(bool), default false to preserve #8715 semantics) that:

• On true: OrdinalAssigner::ensure sequential-fills any missing ordinal and never errors on mismatch, matching pre-General virtual columns support + row numbers as a first use-case #8715 behavior.
• On false (current default): keep the strict check.

Callers that need row-number virtual columns keep the guarantee they rely on; callers that just want to read parquet get the wire-format tolerance that used to be default.

Alternative: downgrade the hard error to a tracing::warn! unless the caller enables row-number virtual columns. This flips the default back to tolerant but keeps the invariant for the code path that actually depends on it.

I'm happy to submit the PR if the direction is agreed. Wanted to open an issue first because this affects the semantics of the default reader path.

References

• Introducing PR: General virtual columns support + row numbers as a first use-case #8715
• Motivating issue: [Parquet] Support file row number in Parquet reader #7299
• Row-group ordinal field: RowGroup.ordinal in parquet.thrift, marked optional

[zhuqi-lucas]

take

[alamb]

It looks like this is the relevant conversation on #8715: https://github.com/apache/arrow-rs/pull/8715/changes#r2499219469

I don't think there is a reason we have to error if some row groups have ordinal assigned and some do not.

From my reading of the spec, there is no requirement that the ordinal field is either all set or all not set:
https://github.com/apache/parquet-format/blob/21ecc2b0f5786070561ae0d34152230853c83e2b/src/main/thrift/parquet.thrift#L1054-L1055

Alternative: downgrade the hard error to a tracing::warn! unless the caller enables row-number virtual columns. This flips the default back to tolerant but keeps the invariant for the code path that actually depends on it.

I think this is better (without the warn!).

So specifically I suggest

1. assigning ordinal to any row group that doesn't have it

I am not quite sure how to handle metadata where the ordinal is not correctly set (e.g. ordinal=10 even when there is only 1 row groups0. We could simply overwrite the oridinal value in the Metadata to be consistent, on the theory that we should be as lenient as posisble on reads and strict on writes

Maybe @vustef remembers more context

Maybe @etseidl has an opinion on this matter

[etseidl]

This also ties in with #10149.

We also have to keep encryption in mind, since the ordinal is used for finding chunk keys IIUC. I'm fine with not assigning the ordinal unless a) it's set in the metadata or b) row numbers are enabled. We're coming up on 69.0.0 soon anyway, so I don't think we need a solution that keeps the current behavior.

[zhuqi-lucas]

Thanks @alamb @etseidl for the quick feedback!

Concretely I plan:

• OrdinalAssigner::ensure no longer errors on mixed-ordinal metadata.
• Ordinals are assigned only when needed for row numbering (or already set in metadata); no gratuitous rewrites.
• Row-number virtual column callers keep their strict guarantee via the existing enablement path.

Will open a PR soon.

[vustef]

Thanks all, the plan sounds good to me as well. RowNumberReader will error out in case missing ordinals, which will include mixed row-groups, so erroring out in OrdinalAssigner is not needed.

The only caveat, and I forgot how things actually work, but if it works by reusing deserialized data, then this might be a problem:

1. Use a reader without row number. It's a file without rowgroup ordinal, on all row groups. OrdinalAssigner is skipped.
2. Now reuse the same metadata, but with row number virtual column. This fails, because there are no row group ordinals. It would've failed if this was the first read that deserialized metadata.

So perhaps this constraint should be lifted:

Ordinals are assigned only when needed for row numbering (or already set in metadata)

It's enough that it doesn't error out, but we could keep ordinal assigning the best effort regardless of whether row numbering is used?

[zhuqi-lucas]

Thanks @vustef! Good catch on the metadata-reuse case — that would be a nasty state-dependent bug where the same "read with row numbers" call succeeds or fails depending on whether the metadata was decoded fresh or reused from a prior no-row-number read.

Agreed, best-effort sequential-fill on read is the right shape:

• If RowGroup.ordinal is set in metadata → preserve it (needed for encryption chunk key lookup).
• If missing → sequential-fill unconditionally, regardless of whether row numbers are enabled.
• Never error on mixed/missing ordinals.

This gives us the "lenient on reads, strict on writes" behavior consistently, and any downstream consumer (row numbers, encryption, anything future) sees complete ordinals no matter how the metadata was obtained.

[vustef]

One other thing came to mind @zhuqi-lucas . We probably need to be careful about mixed row group ordinals case. What happens if we assign ordinals to some row groups but not to others, and user then uses numbering?

They might be doing row group elimination and therefore accidentally skipping row groups without ordinals. Then another query comes, this time it reads a row group without ordinal, and it fails.

I think we want uniform behaviour in case of mixed row group ordinals - row numbering should never work. Which means once we encounter a row group without ordinal, while having previously encountered ordinals, we still want row numbering not to work ever.

[zhuqi-lucas]

Good catch @vustef. Sequential-fill assumes we can see all row groups, and row-group pruning would break that under mixed metadata — silent stale numbers are worse than a clean failure.

Refining the plan:

• All ordinals present → honor them.
• No ordinals present → sequential-fill.
• Mixed → leave metadata untouched; row numbering does not work for this file (deterministic across queries, regardless of pruning).

Non-row-number reads succeed in all three cases — that was the original bug.

[zhuqi-lucas]

Opened #10449 implementing the plan above (all present → honor / none → unconditional sequential-fill / mixed → untouched + row numbering fails deterministically). PTAL @alamb @etseidl @vustef