Skip to content

[GH-3609]: Add new sort order for int96 timestamps - #3610

Open
divjotarora wants to merge 2 commits into
apache:masterfrom
divjotarora:int96-stats
Open

[GH-3609]: Add new sort order for int96 timestamps#3610
divjotarora wants to merge 2 commits into
apache:masterfrom
divjotarora:int96-stats

Conversation

@divjotarora

@divjotarora divjotarora commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Rationale for this change

See parquet-format proposal for rationale: apache/parquet-format#584

What changes are included in this PR?

Add new INT96_TIMESTAMP_ORDER sort order and related parsing.

Are these changes tested?

Yes, new unit tests.

Are there any user-facing changes?

Two new flags and a Thrift change.

Closes #3609

@divjotarora divjotarora mentioned this pull request Jun 11, 2026
Merged
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/ColumnOrder.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveType.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/InternalParquetRecordWriter.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java Outdated
rdblue
rdblue reviewed Jun 11, 2026
View reviewed changes
Comment thread parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputFormat.java Outdated
@divjotarora
divjotarora requested a review from rdblue June 15, 2026 19:20
@divjotarora divjotarora mentioned this pull request Jun 23, 2026
Draft
@wgtmac

wgtmac commented Jun 25, 2026

Copy link
Copy Markdown
Member

Please resolve the conflicts.

@divjotarora

divjotarora commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Please resolve the conflicts.

@wgtmac Sorry for the delay, I was on holiday last week. Conflicts are resolved now

rdblue
rdblue reviewed Jul 10, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveType.java Outdated
rdblue
rdblue reviewed Jul 10, 2026
View reviewed changes
Comment thread parquet-column/src/test/java/org/apache/parquet/schema/TestPrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jul 10, 2026
View reviewed changes
Comment thread parquet-column/src/test/java/org/apache/parquet/schema/TestPrimitiveComparator.java Outdated
@rdblue

rdblue commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

I think there are a few things to be fixed, but overall I trust that this works.

@divjotarora
divjotarora requested a review from rdblue July 13, 2026 16:51
rdblue
rdblue reviewed Jul 14, 2026
View reviewed changes
Comment thread parquet-column/src/main/java/org/apache/parquet/schema/PrimitiveComparator.java Outdated
rdblue
rdblue reviewed Jul 14, 2026
View reviewed changes
Comment thread parquet-column/src/test/java/org/apache/parquet/schema/TestPrimitiveComparator.java Outdated
@divjotarora
divjotarora requested a review from rdblue July 15, 2026 11:00
@rdblue

rdblue commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

This looks good to me, but there are CI failures from spotless. When CI is passing, I'll commit this.

@divjotarora

divjotarora commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

This looks good to me, but there are CI failures from spotless. When CI is passing, I'll commit this.

Fixed the spotless checks + the merge conflicts from the master branch

@divjotarora

divjotarora commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Note: merging this is blocked because a parquet-format release containing apache/parquet-format#584 hasn't been cut yet, so we cannot pull a parquet.thrift definition with this change and the linked CI jobs fail during compilation.

Jiayi-Wang-db added a commit to Jiayi-Wang-db/parquet-java that referenced this pull request Jul 31, 2026
@Jiayi-Wang-db
PARQUET-2249: Write IEEE 754 total order by default for floating-poin…
0bc423d 
…t columns

Follow-up to apache#3393, which added IEEE_754_TOTAL_ORDER support but kept
TYPE_DEFINED_ORDER as the default for FLOAT, DOUBLE and FLOAT16 columns.

Keeping the type-defined order as the default is a latent backward-compat
hazard: the writer now computes finite min/max over the non-NaN subset and
records nan_count, but an old reader that predates nan_count ignores it,
accepts the finite bounds, and can incorrectly prune row groups that contain
NaN. Readers instead ignore statistics written under an unknown sort order, so
writing IEEE 754 total order by default is the safer behavior. See the
discussion on apache#3393.

This makes FLOAT, DOUBLE and FLOAT16 columns built without an explicit column
order default to IEEE_754_TOTAL_ORDER (mirroring how apache#3610 defaults INT96 to
INT96_TIMESTAMP_ORDER). Columns with a logical annotation that does not accept
IEEE 754 total order (e.g. an unknown annotation) fall back to type-defined
order so they remain constructible.

To stay backward compatible on read, a footer that carries no column_orders
list predates IEEE_754_TOTAL_ORDER, so floating-point columns read from such a
footer are given type-defined order rather than inheriting the new
construction-time default; their legacy statistics are thus not reinterpreted
under IEEE 754 total order.

Tests that exercise the legacy type-defined NaN / +-0 semantics are pinned to
TYPE_DEFINED_ORDER explicitly, and new converter tests cover the default
serialization and the column-order-less read path.

Co-authored-by: Isaac
Jiayi-Wang-db added a commit to Jiayi-Wang-db/parquet-java that referenced this pull request Jul 31, 2026
@Jiayi-Wang-db
PARQUET-2249: Write IEEE 754 total order by default for floating-poin…
9388a3e 
…t columns

Follow-up to apache#3393, which added IEEE_754_TOTAL_ORDER support but kept
TYPE_DEFINED_ORDER as the default for FLOAT, DOUBLE and FLOAT16 columns.

Keeping the type-defined order as the default is a latent backward-compat
hazard: the writer now computes finite min/max over the non-NaN subset and
records nan_count, but an old reader that predates nan_count ignores it,
accepts the finite bounds, and can incorrectly prune row groups that contain
NaN. Readers instead ignore statistics written under an unknown sort order, so
writing IEEE 754 total order by default is the safer behavior. See the
discussion on apache#3393.

This makes FLOAT, DOUBLE and FLOAT16 columns built without an explicit column
order default to IEEE_754_TOTAL_ORDER (mirroring how apache#3610 defaults INT96 to
INT96_TIMESTAMP_ORDER). Columns with a logical annotation that does not accept
IEEE 754 total order (e.g. an unknown annotation) fall back to type-defined
order so they remain constructible.

To stay backward compatible on read, a footer that carries no column_orders
list predates IEEE_754_TOTAL_ORDER, so floating-point columns read from such a
footer are given type-defined order rather than inheriting the new
construction-time default; their legacy statistics are thus not reinterpreted
under IEEE 754 total order.

Tests that exercise the legacy type-defined NaN / +-0 semantics are pinned to
TYPE_DEFINED_ORDER explicitly, and new converter tests cover the default
serialization and the column-order-less read path.

Co-authored-by: Isaac
Jiayi-Wang-db added a commit to Jiayi-Wang-db/parquet-java that referenced this pull request Jul 31, 2026
@Jiayi-Wang-db
PARQUET-2249: Write IEEE 754 total order by default for floating-poin…
38f5429 
…t columns

Follow-up to apache#3393, which added IEEE_754_TOTAL_ORDER support but kept
TYPE_DEFINED_ORDER as the default for FLOAT, DOUBLE and FLOAT16 columns.

Keeping the type-defined order as the default is a latent backward-compat
hazard: the writer now computes finite min/max over the non-NaN subset and
records nan_count, but an old reader that predates nan_count ignores it,
accepts the finite bounds, and can incorrectly prune row groups that contain
NaN. Readers instead ignore statistics written under an unknown sort order, so
writing IEEE 754 total order by default is the safer behavior. See the
discussion on apache#3393.

This makes FLOAT, DOUBLE and FLOAT16 columns built without an explicit column
order default to IEEE_754_TOTAL_ORDER (mirroring how apache#3610 defaults INT96 to
INT96_TIMESTAMP_ORDER). Columns with a logical annotation that does not accept
IEEE 754 total order (e.g. an unknown annotation) fall back to type-defined
order so they remain constructible. The default-order selection is unified in
PrimitiveType.defaultColumnOrder so construction and text serialization agree.

To stay backward compatible on read, a footer that carries no column_orders
list predates IEEE_754_TOTAL_ORDER, so floating-point columns read from such a
footer are given type-defined order rather than inheriting the new
construction-time default; their legacy statistics are thus not reinterpreted
under IEEE 754 total order.

The text schema representation now carries a non-default column order
(columnorder(...) after the type/annotation) and MessageTypeParser parses it,
so a column order set explicitly survives toString()/parse round-trips such as
the one GroupWriteSupport performs. Columns left at their default emit no token,
keeping existing schema strings unchanged.

Tests that exercise the legacy type-defined NaN / +-0 semantics set
TYPE_DEFINED_ORDER explicitly, and new tests cover the default serialization,
the column-order-less read path, and the text round-trip.

Co-authored-by: Isaac
Jiayi-Wang-db added a commit to Jiayi-Wang-db/parquet-java that referenced this pull request Jul 31, 2026
@Jiayi-Wang-db
PARQUET-2249: Write IEEE 754 total order by default for floating-poin…
c8a7179 
…t columns

Follow-up to apache#3393, which added IEEE_754_TOTAL_ORDER support but kept
TYPE_DEFINED_ORDER as the default for FLOAT, DOUBLE and FLOAT16 columns.

Keeping the type-defined order as the default is a latent backward-compat
hazard: the writer now computes finite min/max over the non-NaN subset and
records nan_count, but an old reader that predates nan_count ignores it,
accepts the finite bounds, and can incorrectly prune row groups that contain
NaN. Readers instead ignore statistics written under an unknown sort order, so
writing IEEE 754 total order by default is the safer behavior. See the
discussion on apache#3393.

This makes FLOAT, DOUBLE and FLOAT16 columns built without an explicit column
order default to IEEE_754_TOTAL_ORDER (mirroring how apache#3610 defaults INT96 to
INT96_TIMESTAMP_ORDER). Columns with a logical annotation that does not accept
IEEE 754 total order (e.g. an unknown annotation) fall back to type-defined
order so they remain constructible. The default-order selection is unified in
PrimitiveType.defaultColumnOrder so construction and text serialization agree.

To stay backward compatible on read, a footer that carries no column_orders
list predates IEEE_754_TOTAL_ORDER, so floating-point columns read from such a
footer are given type-defined order rather than inheriting the new
construction-time default; their legacy statistics are thus not reinterpreted
under IEEE 754 total order.

The text schema representation now carries a non-default column order
(columnorder(...) after the type/annotation) and MessageTypeParser parses it,
so a column order set explicitly survives toString()/parse round-trips such as
the one GroupWriteSupport performs. Columns left at their default emit no token,
keeping existing schema strings unchanged.

Tests that exercise the legacy type-defined NaN / +-0 semantics set
TYPE_DEFINED_ORDER explicitly, and new tests cover the default serialization,
the column-order-less read path, and the text round-trip.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rdblue rdblue rdblue approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add new sort order for int96 timestamps

3 participants

@divjotarora @wgtmac @rdblue