From 21fb21133a9f30158447c05747e689f4a53eeb9c Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 4 Jun 2026 16:50:21 -0400 Subject: [PATCH 01/17] Document Parquet Versions --- content/en/docs/File Format/versions.md | 258 ++++++++++++++++++++++++ package.json | 6 +- 2 files changed, 261 insertions(+), 3 deletions(-) create mode 100644 content/en/docs/File Format/versions.md diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md new file mode 100644 index 00000000..8e2001b4 --- /dev/null +++ b/content/en/docs/File Format/versions.md @@ -0,0 +1,258 @@ +--- +title: "Parquet format versions" +linkTitle: "Versions" +weight: 9 +--- + +This page describes how features are added to the [Parquet format +specification](https://github.com/apache/parquet-format) and how they affect +reader and writer compatibility. See the +[Implementation status](../implementationstatus/) page for which implementations +(arrow, parquet-java, arrow-rs, etc.) support each feature. + +*Note*: If you find out-of-date information, please open an issue or pull request. + +## Backwards compatible features + +Backwards compatible features can still be **read by older readers**, though with +a degraded experience: they ignore the new information rather than failing. +Examples: + +* **Bloom filters**: a reader that does not understand them ignores the pruning + metadata but still reads the data correctly. +* **Logical type annotations** such as `VARIANT`: an older reader still reads the + underlying physical column (e.g. `BYTE_ARRAY`) as raw bytes, but does not apply + the logical type. + +## Backwards incompatible features + +Backwards incompatible features make the data **unreadable** to older software +that does not support them. Examples: + +* **New encodings** (e.g. the `DELTA_*` encodings, `BYTE_STREAM_SPLIT`, + `RLE_DICTIONARY`): a reader that does not implement them cannot decode the + column values. +* **Data Page V2 headers**: a reader that only understands `DataPageHeader` + cannot parse `DataPageHeaderV2` pages. + +## `FileMetadata` version field + +Each Parquet file has a `version` field in the [`thrift FileMetadata`] that +declares which features the file may use, and thus what a reader **must** support +to read it. + +**Note**: Many writers set the version field to `1` even for files that use +format 2.0 features, which has caused [confusion and interoperability +issues][closing-out-2.0]. + +## `parquet-format` release versions + +The Thrift definition is released independently of implementations such as +parquet-java or arrow-rs, following the Apache release process and +[semantic versioning]: + +1. The major version corresponds to the [`thrift FileMetadata`] `version` field. +2. Minor releases (e.g. `2.10.0` to `2.11.0`) may add backwards compatible + features, but never breaking ones. The minor version is not recorded in the + file itself. + +## Adding new features + +New features are added by discussion and voting on the [parquet dev mailing list] +(full process [here]). Once approved, a feature is added to the spec and ships in +the next parquet-format release. + +[parquet dev mailing list]: https://lists.apache.org/list.html?dev@parquet.apache.org +[semantic versioning]: https://semver.org/ +[`thrift FileMetadata`]: https://github.com/apache/parquet-format/blob/c42c2cb4ecfccb38153375e24b702a82fd763cc0/src/main/thrift/parquet.thrift#L1365-L1373 +[here]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format +[closing-out-2.0]: https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2 + +## Backwards incompatible features by version + +Backwards incompatible features and the format version each became available in: + +* **V1**: the original Parquet format (1.0). +* **V2**: format version 2.0. + +| Feature | V1 | V2 | Released in | Source | Notes | +| ------------------------------------------ | ---- | ---- | ----------------------------- | --- | ------------------------- | +| [BOOLEAN] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT32] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT64] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT96 (deprecated)] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [FLOAT] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [DOUBLE] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BYTE_ARRAY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [FIXED_LEN_BYTE_ARRAY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Data Page V1] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Data Page V2] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | +| [PLAIN] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [PLAIN_DICTIONARY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [RLE] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BIT_PACKED (deprecated)] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [RLE_DICTIONARY] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_BINARY_PACKED] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_LENGTH_BYTE_ARRAY] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_BYTE_ARRAY] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | +| [BYTE_STREAM_SPLIT] | | ✅ | [2.8.0] | [2.7.0..2.8.0] | [Approved 2019-12-03] | +| [BYTE_STREAM_SPLIT
(Additional Types)] | | ✅ | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-03-18] | +| [UNCOMPRESSED] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [SNAPPY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [GZIP] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [LZO] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BROTLI] | | ✅ | [2.4.0] | [2.3.1..2.4.0] | | +| [LZ4 (deprecated)] | | ✅ | [2.4.0] | [2.3.1..2.4.0] | | +| [LZ4_RAW] | | ✅ | [2.9.0] | [2.8.0..2.9.0] | | +| [ZSTD] | | ✅ | [2.4.0] | [2.3.1..2.4.0] | | +| [Modular encryption] | | ✅ | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-01-16] | + + +> **Note:** Compression codecs and modular encryption are not gated by the +> `version` field — a reader needs support for the specific codec or for +> encryption regardless of version. + +## Backwards compatible additions + +Files written with these features can be read by all readers, but older +readers may not understand the new information. + +| Feature | Released in | Source | Notes | +| ------------------------------------------- | ----------------------------- | --- | ------------------------- | +| [xxHash-based bloom filters] | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-09-09] | +| [Bloom filter length] | [2.10.0] | [2.9.0..2.10.0] | | +| [Page index] | [2.4.0] | [2.3.1..2.4.0] | | +| [Page CRC32 checksum] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Size statistics] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-11-14] | +| [Geospatial statistics] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [Binary protocol extensions] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-09-06] | +| [IEEE 754 total order and NaN counts] | not yet released | [#514] | [Approved 2026-05-26] | +| [STRING] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [ENUM] | [2.0.0] | [1.0.0..2.0.0] | | +| [UUID] | [2.6.0] | [2.5.0..2.6.0] | | +| [Signed and unsigned integer logical types] | [2.2.0] | [2.1.0..2.2.0] | | +| [DECIMAL (INT32)] | [2.1.0] | [2.0.0..2.1.0] | | +| [DECIMAL (INT64)] | [2.1.0] | [2.0.0..2.1.0] | | +| [DECIMAL (BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | +| [DECIMAL (FIXED_LEN_BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | +| [FLOAT16] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-10-13] | +| [DATE] | [2.2.0] | [2.1.0..2.2.0] | | +| [TIME (INT32)] | [2.2.0] | [2.1.0..2.2.0] | | +| [TIME (INT64)] | [2.4.0] | [2.3.1..2.4.0] | | +| [TIMESTAMP (INT64)] | [2.2.0] | [2.1.0..2.2.0] | | +| [INTERVAL] | [2.2.0] | [2.1.0..2.2.0] | | +| [JSON] | [2.2.0] | [2.1.0..2.2.0] | | +| [BSON] | [2.2.0] | [2.1.0..2.2.0] | | +| [VARIANT] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | +| [Variant shredding] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | +| [GEOMETRY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [GEOGRAPHY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [LIST] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [MAP] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [UNKNOWN (always null)] | [2.4.0] | [2.3.1..2.4.0] | | + +[PLAIN]: https://github.com/apache/parquet-format/blob/master/Encodings.md#plain-plain--0 +[PLAIN_DICTIONARY]: https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8 +[RLE]: https://github.com/apache/parquet-format/blob/master/Encodings.md#run-length-encoding--bit-packing-hybrid-rle--3 +[RLE_DICTIONARY]: https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8 +[DELTA_BINARY_PACKED]: https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5 +[DELTA_LENGTH_BYTE_ARRAY]: https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-length-byte-array-delta_length_byte_array--6 +[DELTA_BYTE_ARRAY]: https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-strings-delta_byte_array--7 +[BYTE_STREAM_SPLIT]: https://github.com/apache/parquet-format/blob/master/Encodings.md#byte-stream-split-byte_stream_split--9 +[Modular encryption]: https://github.com/apache/parquet-format/blob/master/Encryption.md +[xxHash-based bloom filters]: https://github.com/apache/parquet-format/blob/master/BloomFilter.md +[Page index]: https://github.com/apache/parquet-format/blob/master/PageIndex.md +[FLOAT16]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#float16 +[VARIANT]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md +[GEOMETRY]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geometry +[GEOGRAPHY]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geography + +[1.0.0]: https://github.com/apache/parquet-format/releases/tag/parquet-format-1.0.0 +[2.0.0]: https://github.com/apache/parquet-format/releases/tag/parquet-format-2.0.0 +[2.8.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.8.0 +[2.11.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.11.0 +[2.4.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.4.0 +[2.9.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.9.0 +[2.7.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.7.0 +[2.10.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.10.0 +[2.12.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.12.0 + +[Approved 2019-12-03]: https://lists.apache.org/thread/xs5qt2odm299pxgqb22mty2csc1so5yr +[Approved 2024-03-18]: https://lists.apache.org/thread/nlsj0ftxy7y4ov1678rgy5zc7dmogg6q +[Approved 2019-01-16]: https://lists.apache.org/thread/l8zcwnbrnhjh3w2k1lyb0v6ct5lnzr0h +[Approved 2019-09-09]: https://lists.apache.org/thread/ktdx1xp0d2gjfgkcvd29zxvt3cgg88bo +[Approved 2023-11-14]: https://lists.apache.org/thread/wgobz41mfldbhqpg9q4mdwypghg2cxg2 +[Approved 2023-10-13]: https://lists.apache.org/thread/gyvqcx9ssxkjlrwogqwy7n4z6ofdm871 +[Approved 2025-08-24]: https://lists.apache.org/thread/obn1yzhgm5zlznwrdpg7f66mswwooxw7 +[Approved 2025-02-09]: https://lists.apache.org/thread/s6s714c98cn9gg22mnk5nsn7xymym8xo + +[1.0.0..2.0.0]: https://github.com/apache/parquet-format/compare/parquet-format-1.0.0...parquet-format-2.0.0 +[2.7.0..2.8.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.7.0...apache-parquet-format-2.8.0 +[2.10.0..2.11.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.10.0...apache-parquet-format-2.11.0 +[2.3.1..2.4.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.3.1...apache-parquet-format-2.4.0 +[2.8.0..2.9.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.8.0...apache-parquet-format-2.9.0 +[2.6.0..2.7.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.6.0...apache-parquet-format-2.7.0 +[2.9.0..2.10.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.9.0...apache-parquet-format-2.10.0 +[2.11.0..2.12.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.11.0...apache-parquet-format-2.12.0 + +[2.1.0]: https://github.com/apache/parquet-format/releases/tag/parquet-format-2.1.0 +[2.2.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.2.0 +[2.6.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.6.0 +[2.0.0..2.1.0]: https://github.com/apache/parquet-format/compare/parquet-format-2.0.0...parquet-format-2.1.0 +[2.1.0..2.2.0]: https://github.com/apache/parquet-format/compare/parquet-format-2.1.0...apache-parquet-format-2.2.0 +[2.5.0..2.6.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.5.0...apache-parquet-format-2.6.0 + +[tree-1.0.0]: https://github.com/apache/parquet-format/tree/parquet-format-1.0.0 + +[Variant shredding]: https://github.com/apache/parquet-format/blob/master/VariantShredding.md +[Geospatial statistics]: https://github.com/apache/parquet-format/blob/master/Geospatial.md#statistics +[Binary protocol extensions]: https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md +[#514]: https://github.com/apache/parquet-format/pull/514 +[Approved 2024-09-06]: https://lists.apache.org/thread/x3472kldrq5kjnld9ztj1jozz25f40hg +[Approved 2026-05-26]: https://lists.apache.org/thread/h0k0hqo0sojqphnbnrkp8b0gmwdzq9on + +[Bloom filter length]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L933 +[Page CRC32 checksum]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L829 +[Size statistics]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L202 +[IEEE 754 Column Order]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1061 +[STRING]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string +[ENUM]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#enum +[UUID]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#uuid +[8, 16, 32, 64 bit signed and unsigned INT]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers +[DECIMAL (INT32)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal +[DECIMAL (INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal +[DECIMAL (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal +[DECIMAL (FIXED_LEN_BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal +[DATE]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#date +[TIME (INT32)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time +[TIME (INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time +[TIMESTAMP (INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#timestamp +[INTERVAL]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval +[JSON]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#json +[BSON]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#bson +[LIST]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists +[MAP]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps +[UNKNOWN (always null)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#unknown-always-null +[Signed and unsigned integer logical types]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers +[IEEE 754 total order and NaN counts]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1061 + +[BOOLEAN]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L33 +[INT32]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L34 +[INT64]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L35 +[INT96 (deprecated)]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L36 +[FLOAT]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L37 +[DOUBLE]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L38 +[BYTE_ARRAY]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L39 +[FIXED_LEN_BYTE_ARRAY]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L40 +[Data Page V1]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L671 +[Data Page V2]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L724 +[BIT_PACKED (deprecated)]: https://github.com/apache/parquet-format/blob/master/Encodings.md#bit-packed-deprecated-bit_packed--4 +[BYTE_STREAM_SPLIT
(Additional Types)]: https://github.com/apache/parquet-format/blob/master/Encodings.md#byte-stream-split-byte_stream_split--9 +[UNCOMPRESSED]: https://github.com/apache/parquet-format/blob/master/Compression.md#uncompressed +[SNAPPY]: https://github.com/apache/parquet-format/blob/master/Compression.md#snappy +[GZIP]: https://github.com/apache/parquet-format/blob/master/Compression.md#gzip +[LZO]: https://github.com/apache/parquet-format/blob/master/Compression.md#lzo +[BROTLI]: https://github.com/apache/parquet-format/blob/master/Compression.md#brotli +[LZ4 (deprecated)]: https://github.com/apache/parquet-format/blob/master/Compression.md#lz4 +[LZ4_RAW]: https://github.com/apache/parquet-format/blob/master/Compression.md#lz4_raw +[ZSTD]: https://github.com/apache/parquet-format/blob/master/Compression.md#zstd diff --git a/package.json b/package.json index 91825029..d5e65f5e 100644 --- a/package.json +++ b/package.json @@ -18,9 +18,9 @@ "homepage": "https://github.com/apache/parquet-site#readme", "devDependencies": { "@mermaid-js/mermaid-cli": "^11.4.2", - "autoprefixer": "^10.4.18", + "autoprefixer": "^10.5.0", "bootstrap": "^5.3.6", - "postcss": "^8.4.35", - "postcss-cli": "^11.0.0" + "postcss": "^8.5.15", + "postcss-cli": "^11.0.1" } } From 430ffcebca93e12b8b42b78572bb5a9fa286a3b3 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Fri, 5 Jun 2026 13:02:23 -0400 Subject: [PATCH 02/17] Update content/en/docs/File Format/versions.md Co-authored-by: Fokko Driesprong --- content/en/docs/File Format/versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 8e2001b4..824bb307 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -1,6 +1,6 @@ --- title: "Parquet format versions" -linkTitle: "Versions" +linkTitle: "Format versions" weight: 9 --- From 7f37c8e25352ee3282b70ee7d149a33911495e5c Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Fri, 5 Jun 2026 13:18:48 -0400 Subject: [PATCH 03/17] Use forward compatibile terminology --- content/en/docs/File Format/versions.md | 39 +++++++++++++++---------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 824bb307..f2b8a41c 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -12,19 +12,24 @@ reader and writer compatibility. See the *Note*: If you find out-of-date information, please open an issue or pull request. -## Backwards compatible features +## Feature compatibility -Backwards compatible features can still be **read by older readers**, though with -a degraded experience: they ignore the new information rather than failing. -Examples: +The Parquet format spec [classifies changes] by their effect on reader and +writer compatibility. Changes differ in their *forward* compatibility — whether +an older reader can read files that use a newer feature. + +**Forward compatible** features remain **readable by older readers**, with a +possibly degraded experience: some metadata may be missing or performance may +suffer, but the reader does not fail. Examples: -* **Bloom filters**: a reader that does not understand them ignores the pruning - metadata but still reads the data correctly. -* **Logical type annotations** such as `VARIANT`: an older reader still reads the - underlying physical column (e.g. `BYTE_ARRAY`) as raw bytes, but does not apply +* **Bloom filters**: a reader that ignores them skips the pruning metadata but + still reads the data correctly. +* **Logical type annotations** such as `VARIANT`: an older reader reads the + underlying physical column (e.g. `BYTE_ARRAY`) as raw bytes without applying the logical type. -## Backwards incompatible features +**Forward incompatible** features make the data **unreadable** to older software. +Examples: Backwards incompatible features make the data **unreadable** to older software that does not support them. Examples: @@ -35,6 +40,8 @@ that does not support them. Examples: * **Data Page V2 headers**: a reader that only understands `DataPageHeader` cannot parse `DataPageHeaderV2` pages. +[classifies changes]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement + ## `FileMetadata` version field Each Parquet file has a `version` field in the [`thrift FileMetadata`] that @@ -52,8 +59,8 @@ parquet-java or arrow-rs, following the Apache release process and [semantic versioning]: 1. The major version corresponds to the [`thrift FileMetadata`] `version` field. -2. Minor releases (e.g. `2.10.0` to `2.11.0`) may add backwards compatible - features, but never breaking ones. The minor version is not recorded in the +2. Minor releases (e.g. `2.10.0` to `2.11.0`) may add compatible + features, but never incompatible ones. The minor version is not recorded in the file itself. ## Adding new features @@ -68,9 +75,9 @@ the next parquet-format release. [here]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format [closing-out-2.0]: https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2 -## Backwards incompatible features by version +## Forward incompatible features by version -Backwards incompatible features and the format version each became available in: +Forward incompatible features and the format version each became available in: * **V1**: the original Parquet format (1.0). * **V2**: format version 2.0. @@ -112,10 +119,10 @@ Backwards incompatible features and the format version each became available in: > `version` field — a reader needs support for the specific codec or for > encryption regardless of version. -## Backwards compatible additions +## Forward compatible additions -Files written with these features can be read by all readers, but older -readers may not understand the new information. +Older readers can read files that use these features but may not understand the +new information. | Feature | Released in | Source | Notes | | ------------------------------------------- | ----------------------------- | --- | ------------------------- | From 3af9dfb4aea3c374ad633e740b5174b1bd472f71 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Fri, 5 Jun 2026 14:06:10 -0400 Subject: [PATCH 04/17] document LogicalType union, and Nano timestamp --- content/en/docs/File Format/versions.md | 70 +++++++++++++------------ 1 file changed, 37 insertions(+), 33 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index f2b8a41c..7853be18 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -124,39 +124,41 @@ Forward incompatible features and the format version each became available in: Older readers can read files that use these features but may not understand the new information. -| Feature | Released in | Source | Notes | -| ------------------------------------------- | ----------------------------- | --- | ------------------------- | -| [xxHash-based bloom filters] | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-09-09] | -| [Bloom filter length] | [2.10.0] | [2.9.0..2.10.0] | | -| [Page index] | [2.4.0] | [2.3.1..2.4.0] | | -| [Page CRC32 checksum] | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [Size statistics] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-11-14] | -| [Geospatial statistics] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | -| [Binary protocol extensions] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-09-06] | -| [IEEE 754 total order and NaN counts] | not yet released | [#514] | [Approved 2026-05-26] | -| [STRING] | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [ENUM] | [2.0.0] | [1.0.0..2.0.0] | | -| [UUID] | [2.6.0] | [2.5.0..2.6.0] | | -| [Signed and unsigned integer logical types] | [2.2.0] | [2.1.0..2.2.0] | | -| [DECIMAL (INT32)] | [2.1.0] | [2.0.0..2.1.0] | | -| [DECIMAL (INT64)] | [2.1.0] | [2.0.0..2.1.0] | | -| [DECIMAL (BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | -| [DECIMAL (FIXED_LEN_BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | -| [FLOAT16] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-10-13] | -| [DATE] | [2.2.0] | [2.1.0..2.2.0] | | -| [TIME (INT32)] | [2.2.0] | [2.1.0..2.2.0] | | -| [TIME (INT64)] | [2.4.0] | [2.3.1..2.4.0] | | -| [TIMESTAMP (INT64)] | [2.2.0] | [2.1.0..2.2.0] | | -| [INTERVAL] | [2.2.0] | [2.1.0..2.2.0] | | -| [JSON] | [2.2.0] | [2.1.0..2.2.0] | | -| [BSON] | [2.2.0] | [2.1.0..2.2.0] | | -| [VARIANT] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | -| [Variant shredding] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | -| [GEOMETRY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | -| [GEOGRAPHY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | -| [LIST] | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [MAP] | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [UNKNOWN (always null)] | [2.4.0] | [2.3.1..2.4.0] | | +| Feature | Released in | Source | Notes | +| ------------------------------------------- | ----------------------------- | --- |-----------------------------------------------------------| +| [xxHash-based bloom filters] | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-09-09] | +| [Bloom filter length] | [2.10.0] | [2.9.0..2.10.0] | | +| [Page index] | [2.4.0] | [2.3.1..2.4.0] | | +| [Page CRC32 checksum] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Size statistics] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-11-14] | +| [Geospatial statistics] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [Binary protocol extensions] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-09-06] | +| [IEEE 754 total order and NaN counts] | not yet released | [#514] | [Approved 2026-05-26] | +| [LogicalType union] | [2.4.0] | [2.3.1..2.4.0] | Supersedes `ConvertedType` enum
deprecated in [2.9.0] | +| [STRING] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [ENUM] | [2.0.0] | [1.0.0..2.0.0] | | +| [UUID] | [2.6.0] | [2.5.0..2.6.0] | | +| [Signed and unsigned integer logical types] | [2.2.0] | [2.1.0..2.2.0] | | +| [DECIMAL (INT32)] | [2.1.0] | [2.0.0..2.1.0] | | +| [DECIMAL (INT64)] | [2.1.0] | [2.0.0..2.1.0] | | +| [DECIMAL (BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | +| [DECIMAL (FIXED_LEN_BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | +| [FLOAT16] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-10-13] | +| [DATE] | [2.2.0] | [2.1.0..2.2.0] | | +| [TIME (INT32)] | [2.2.0] | [2.1.0..2.2.0] | | +| [TIME (INT64)] | [2.4.0] | [2.3.1..2.4.0] | | +| [TIMESTAMP (INT64)] | [2.2.0] | [2.1.0..2.2.0] | | +| [Nanosecond TIME/TIMESTAMP] | [2.6.0] | [2.5.0..2.6.0] | | +| [INTERVAL] | [2.2.0] | [2.1.0..2.2.0] | | +| [JSON] | [2.2.0] | [2.1.0..2.2.0] | | +| [BSON] | [2.2.0] | [2.1.0..2.2.0] | | +| [VARIANT] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | +| [Variant shredding] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | +| [GEOMETRY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [GEOGRAPHY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [LIST] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [MAP] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [UNKNOWN (always null)] | [2.4.0] | [2.3.1..2.4.0] | | [PLAIN]: https://github.com/apache/parquet-format/blob/master/Encodings.md#plain-plain--0 [PLAIN_DICTIONARY]: https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8 @@ -222,6 +224,8 @@ new information. [Page CRC32 checksum]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L829 [Size statistics]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L202 [IEEE 754 Column Order]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1061 +[LogicalType union]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L471 +[Nanosecond TIME/TIMESTAMP]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L352 [STRING]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string [ENUM]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#enum [UUID]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#uuid From 390bad3334d8ac526dced601518fd4b4ace7ec3b Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Fri, 5 Jun 2026 14:15:04 -0400 Subject: [PATCH 05/17] Apply suggestion from @etseidl Co-authored-by: Ed Seidl --- content/en/docs/File Format/versions.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 7853be18..e5a0a7ff 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -31,9 +31,6 @@ suffer, but the reader does not fail. Examples: **Forward incompatible** features make the data **unreadable** to older software. Examples: -Backwards incompatible features make the data **unreadable** to older software -that does not support them. Examples: - * **New encodings** (e.g. the `DELTA_*` encodings, `BYTE_STREAM_SPLIT`, `RLE_DICTIONARY`): a reader that does not implement them cannot decode the column values. From c438950df167d802c1fc0cb78f65f8f28650e2ef Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Fri, 5 Jun 2026 14:20:58 -0400 Subject: [PATCH 06/17] Note PARE footer --- content/en/docs/File Format/versions.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 7853be18..c27c00fe 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -31,9 +31,6 @@ suffer, but the reader does not fail. Examples: **Forward incompatible** features make the data **unreadable** to older software. Examples: -Backwards incompatible features make the data **unreadable** to older software -that does not support them. Examples: - * **New encodings** (e.g. the `DELTA_*` encodings, `BYTE_STREAM_SPLIT`, `RLE_DICTIONARY`): a reader that does not implement them cannot decode the column values. @@ -115,9 +112,10 @@ Forward incompatible features and the format version each became available in: | [Modular encryption] | | ✅ | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-01-16] | -> **Note:** Compression codecs and modular encryption are not gated by the -> `version` field — a reader needs support for the specific codec or for -> encryption regardless of version. +> **Note:** Files with an [encrypted footer] use different magic bytes (`PARE` +> instead of `PAR1`), making it clear to readers they must support modular +> encryption to read the file; [plaintext footer] files use `PAR1` so legacy +> readers can still read their unencrypted columns. ## Forward compatible additions @@ -169,6 +167,8 @@ new information. [DELTA_BYTE_ARRAY]: https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-strings-delta_byte_array--7 [BYTE_STREAM_SPLIT]: https://github.com/apache/parquet-format/blob/master/Encodings.md#byte-stream-split-byte_stream_split--9 [Modular encryption]: https://github.com/apache/parquet-format/blob/master/Encryption.md +[encrypted footer]: https://github.com/apache/parquet-format/blob/master/Encryption.md#54-encrypted-footer-mode +[plaintext footer]: https://github.com/apache/parquet-format/blob/master/Encryption.md#55-plaintext-footer-mode [xxHash-based bloom filters]: https://github.com/apache/parquet-format/blob/master/BloomFilter.md [Page index]: https://github.com/apache/parquet-format/blob/master/PageIndex.md [FLOAT16]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#float16 From 45b2b9493a1f61ca4702c185ea7ec8c0ae5dc2a7 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Fri, 5 Jun 2026 14:28:29 -0400 Subject: [PATCH 07/17] Consistently list physical types with logical types --- content/en/docs/File Format/versions.md | 46 ++++++++++++------------- 1 file changed, 22 insertions(+), 24 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index c27c00fe..90919ba7 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -133,27 +133,27 @@ new information. | [Binary protocol extensions] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-09-06] | | [IEEE 754 total order and NaN counts] | not yet released | [#514] | [Approved 2026-05-26] | | [LogicalType union] | [2.4.0] | [2.3.1..2.4.0] | Supersedes `ConvertedType` enum
deprecated in [2.9.0] | -| [STRING] | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [ENUM] | [2.0.0] | [1.0.0..2.0.0] | | -| [UUID] | [2.6.0] | [2.5.0..2.6.0] | | -| [Signed and unsigned integer logical types] | [2.2.0] | [2.1.0..2.2.0] | | +| [STRING (BYTE_ARRAY)] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [ENUM (BYTE_ARRAY)] | [2.0.0] | [1.0.0..2.0.0] | | +| [UUID (FIXED_LEN_BYTE_ARRAY(16))] | [2.6.0] | [2.5.0..2.6.0] | | +| [Signed and unsigned integer logical types (INT32, INT64)] | [2.2.0] | [2.1.0..2.2.0] | | | [DECIMAL (INT32)] | [2.1.0] | [2.0.0..2.1.0] | | | [DECIMAL (INT64)] | [2.1.0] | [2.0.0..2.1.0] | | | [DECIMAL (BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | | [DECIMAL (FIXED_LEN_BYTE_ARRAY)] | [2.1.0] | [2.0.0..2.1.0] | | -| [FLOAT16] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-10-13] | -| [DATE] | [2.2.0] | [2.1.0..2.2.0] | | +| [FLOAT16 (FIXED_LEN_BYTE_ARRAY(2))] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-10-13] | +| [DATE (INT32)] | [2.2.0] | [2.1.0..2.2.0] | | | [TIME (INT32)] | [2.2.0] | [2.1.0..2.2.0] | | | [TIME (INT64)] | [2.4.0] | [2.3.1..2.4.0] | | | [TIMESTAMP (INT64)] | [2.2.0] | [2.1.0..2.2.0] | | | [Nanosecond TIME/TIMESTAMP] | [2.6.0] | [2.5.0..2.6.0] | | -| [INTERVAL] | [2.2.0] | [2.1.0..2.2.0] | | -| [JSON] | [2.2.0] | [2.1.0..2.2.0] | | -| [BSON] | [2.2.0] | [2.1.0..2.2.0] | | +| [INTERVAL (FIXED_LEN_BYTE_ARRAY(12))] | [2.2.0] | [2.1.0..2.2.0] | | +| [JSON (BYTE_ARRAY)] | [2.2.0] | [2.1.0..2.2.0] | | +| [BSON (BYTE_ARRAY)] | [2.2.0] | [2.1.0..2.2.0] | | | [VARIANT] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | | [Variant shredding] | [2.12.0] | [2.11.0..2.12.0] | [Approved 2025-08-24] | -| [GEOMETRY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | -| [GEOGRAPHY] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [GEOMETRY (BYTE_ARRAY)] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | +| [GEOGRAPHY (BYTE_ARRAY)] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | | [LIST] | [1.0.0] | [1.0.0][tree-1.0.0] | | | [MAP] | [1.0.0] | [1.0.0][tree-1.0.0] | | | [UNKNOWN (always null)] | [2.4.0] | [2.3.1..2.4.0] | | @@ -171,10 +171,10 @@ new information. [plaintext footer]: https://github.com/apache/parquet-format/blob/master/Encryption.md#55-plaintext-footer-mode [xxHash-based bloom filters]: https://github.com/apache/parquet-format/blob/master/BloomFilter.md [Page index]: https://github.com/apache/parquet-format/blob/master/PageIndex.md -[FLOAT16]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#float16 +[FLOAT16 (FIXED_LEN_BYTE_ARRAY(2))]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#float16 [VARIANT]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md -[GEOMETRY]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geometry -[GEOGRAPHY]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geography +[GEOMETRY (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geometry +[GEOGRAPHY (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#geography [1.0.0]: https://github.com/apache/parquet-format/releases/tag/parquet-format-1.0.0 [2.0.0]: https://github.com/apache/parquet-format/releases/tag/parquet-format-2.0.0 @@ -223,28 +223,26 @@ new information. [Bloom filter length]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L933 [Page CRC32 checksum]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L829 [Size statistics]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L202 -[IEEE 754 Column Order]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1061 [LogicalType union]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L471 [Nanosecond TIME/TIMESTAMP]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L352 -[STRING]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string -[ENUM]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#enum -[UUID]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#uuid -[8, 16, 32, 64 bit signed and unsigned INT]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers +[STRING (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#string +[ENUM (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#enum +[UUID (FIXED_LEN_BYTE_ARRAY(16))]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#uuid [DECIMAL (INT32)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal [DECIMAL (INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal [DECIMAL (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal [DECIMAL (FIXED_LEN_BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal -[DATE]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#date +[DATE (INT32)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#date [TIME (INT32)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time [TIME (INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#time [TIMESTAMP (INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#timestamp -[INTERVAL]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval -[JSON]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#json -[BSON]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#bson +[INTERVAL (FIXED_LEN_BYTE_ARRAY(12))]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval +[JSON (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#json +[BSON (BYTE_ARRAY)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#bson [LIST]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists [MAP]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#maps [UNKNOWN (always null)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#unknown-always-null -[Signed and unsigned integer logical types]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers +[Signed and unsigned integer logical types (INT32, INT64)]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#signed-integers [IEEE 754 total order and NaN counts]: https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L1061 [BOOLEAN]: https://github.com/apache/parquet-format/blob/96656a543a2165d57cc1c9abefaad7f9aeb563a5/src/main/thrift/parquet.thrift#L33 From d8c63bd9be965c76017160950b2f5946e83be47a Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 9 Jun 2026 06:59:16 -0400 Subject: [PATCH 08/17] Clarfiy Semantic Versioning --- content/en/docs/File Format/versions.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 90919ba7..6059d9a0 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -1,6 +1,6 @@ --- title: "Parquet format versions" -linkTitle: "Format versions" +linkTitle: "Features and Versions" weight: 9 --- @@ -52,13 +52,12 @@ issues][closing-out-2.0]. ## `parquet-format` release versions The Thrift definition is released independently of implementations such as -parquet-java or arrow-rs, following the Apache release process and -[semantic versioning]: +parquet-java or arrow-rs, following the Apache release process. Note that +release numbering **DOES NOT FOLLOW** [semantic versioning]: 1. The major version corresponds to the [`thrift FileMetadata`] `version` field. -2. Minor releases (e.g. `2.10.0` to `2.11.0`) may add compatible - features, but never incompatible ones. The minor version is not recorded in the - file itself. +2. Minor releases (e.g. `2.10.0` to `2.11.0`) sometimes contain forward incompatible + features. The minor version is not recorded in the file itself. ## Adding new features From df8c1ca4db3d59901cad243fc794bdb1075c5bb7 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 9 Jun 2026 07:03:08 -0400 Subject: [PATCH 09/17] tweak --- content/en/docs/File Format/versions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 6059d9a0..7d90cd70 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -56,8 +56,8 @@ parquet-java or arrow-rs, following the Apache release process. Note that release numbering **DOES NOT FOLLOW** [semantic versioning]: 1. The major version corresponds to the [`thrift FileMetadata`] `version` field. -2. Minor releases (e.g. `2.10.0` to `2.11.0`) sometimes contain forward incompatible - features. The minor version is not recorded in the file itself. +2. Minor releases (e.g. `2.10.0` to `2.11.0`) sometimes contain forward + incompatible features. The minor version is not recorded in the file itself. ## Adding new features From 89159332dc770c64d88f48fcdeb24be53fc82161 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 9 Jun 2026 09:27:27 -0400 Subject: [PATCH 10/17] Avoid ascribing semantic meaning to the version field --- content/en/docs/File Format/versions.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 7d90cd70..a14a0c1e 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -41,13 +41,10 @@ Examples: ## `FileMetadata` version field -Each Parquet file has a `version` field in the [`thrift FileMetadata`] that -declares which features the file may use, and thus what a reader **must** support -to read it. - -**Note**: Many writers set the version field to `1` even for files that use -format 2.0 features, which has caused [confusion and interoperability -issues][closing-out-2.0]. +Each Parquet file has a `version` field in the [`thrift FileMetadata`]. This +field has historically been used inconsistently: writers populate `1` or `2` +without a consistent relationship to the features actually used. See the +[note in parquet.thrift] and [this discussion][closing-out-2.0] for details. ## `parquet-format` release versions @@ -69,6 +66,7 @@ the next parquet-format release. [semantic versioning]: https://semver.org/ [`thrift FileMetadata`]: https://github.com/apache/parquet-format/blob/c42c2cb4ecfccb38153375e24b702a82fd763cc0/src/main/thrift/parquet.thrift#L1365-L1373 [here]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format +[note in parquet.thrift]: https://github.com/apache/parquet-format/blob/74001e41f5c5a1856b29be115f9c992cab16a4bf/src/main/thrift/parquet.thrift#L1368-L1373 [closing-out-2.0]: https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2 ## Forward incompatible features by version From 0b3a17f0e8cddc39eeccdc3ca2fbd7e2def0b077 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 10 Jun 2026 06:52:38 -0400 Subject: [PATCH 11/17] Remove other mention of V1 and V2 to reduce confusion --- content/en/docs/File Format/versions.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index a14a0c1e..96012303 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -73,9 +73,6 @@ the next parquet-format release. Forward incompatible features and the format version each became available in: -* **V1**: the original Parquet format (1.0). -* **V2**: format version 2.0. - | Feature | V1 | V2 | Released in | Source | Notes | | ------------------------------------------ | ---- | ---- | ----------------------------- | --- | ------------------------- | | [BOOLEAN] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | From 8cb094252a28f8e8b63bd454b4310f50a0e07a61 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 10 Jun 2026 15:44:02 -0400 Subject: [PATCH 12/17] Follow wording from @stseidl to avoid implying semantic meaning to v1/v2 Co-authored-by: Ed Seidl --- content/en/docs/File Format/versions.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 96012303..5cf5d4b6 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -49,12 +49,11 @@ without a consistent relationship to the features actually used. See the ## `parquet-format` release versions The Thrift definition is released independently of implementations such as -parquet-java or arrow-rs, following the Apache release process. Note that +parquet-java or arrow-rs, following the Apache release process. This +release version is not recorded in the FileMetaData. Note that release numbering **DOES NOT FOLLOW** [semantic versioning]: - -1. The major version corresponds to the [`thrift FileMetadata`] `version` field. -2. Minor releases (e.g. `2.10.0` to `2.11.0`) sometimes contain forward - incompatible features. The minor version is not recorded in the file itself. +minor releases (e.g. `2.10.0` to `2.11.0`) sometimes contain forward +incompatible features. ## Adding new features From 1126b539491040f97627f720f9c593a9705a4377 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 10 Jun 2026 15:50:21 -0400 Subject: [PATCH 13/17] remove V1/V2 columns to avoid confusion --- content/en/docs/File Format/versions.md | 62 ++++++++++++------------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 5cf5d4b6..f5bfda86 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -72,37 +72,37 @@ the next parquet-format release. Forward incompatible features and the format version each became available in: -| Feature | V1 | V2 | Released in | Source | Notes | -| ------------------------------------------ | ---- | ---- | ----------------------------- | --- | ------------------------- | -| [BOOLEAN] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [INT32] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [INT64] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [INT96 (deprecated)] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [FLOAT] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [DOUBLE] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [BYTE_ARRAY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [FIXED_LEN_BYTE_ARRAY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [Data Page V1] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [Data Page V2] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | -| [PLAIN] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [PLAIN_DICTIONARY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [RLE] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [BIT_PACKED (deprecated)] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [RLE_DICTIONARY] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | -| [DELTA_BINARY_PACKED] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | -| [DELTA_LENGTH_BYTE_ARRAY] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | -| [DELTA_BYTE_ARRAY] | | ✅ | [2.0.0] | [1.0.0..2.0.0] | | -| [BYTE_STREAM_SPLIT] | | ✅ | [2.8.0] | [2.7.0..2.8.0] | [Approved 2019-12-03] | -| [BYTE_STREAM_SPLIT
(Additional Types)] | | ✅ | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-03-18] | -| [UNCOMPRESSED] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [SNAPPY] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [GZIP] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [LZO] | ✅ | ✅ | [1.0.0] | [1.0.0][tree-1.0.0] | | -| [BROTLI] | | ✅ | [2.4.0] | [2.3.1..2.4.0] | | -| [LZ4 (deprecated)] | | ✅ | [2.4.0] | [2.3.1..2.4.0] | | -| [LZ4_RAW] | | ✅ | [2.9.0] | [2.8.0..2.9.0] | | -| [ZSTD] | | ✅ | [2.4.0] | [2.3.1..2.4.0] | | -| [Modular encryption] | | ✅ | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-01-16] | +| Feature | Released in | Source | Notes | +| ------------------------------------------ | ----------------------------- | --- | ------------------------- | +| [BOOLEAN] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT32] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT64] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [INT96 (deprecated)] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [FLOAT] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [DOUBLE] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BYTE_ARRAY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [FIXED_LEN_BYTE_ARRAY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Data Page V1] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [Data Page V2] | [2.0.0] | [1.0.0..2.0.0] | | +| [PLAIN] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [PLAIN_DICTIONARY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [RLE] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BIT_PACKED (deprecated)] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [RLE_DICTIONARY] | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_BINARY_PACKED] | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_LENGTH_BYTE_ARRAY] | [2.0.0] | [1.0.0..2.0.0] | | +| [DELTA_BYTE_ARRAY] | [2.0.0] | [1.0.0..2.0.0] | | +| [BYTE_STREAM_SPLIT] | [2.8.0] | [2.7.0..2.8.0] | [Approved 2019-12-03] | +| [BYTE_STREAM_SPLIT
(Additional Types)] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-03-18] | +| [UNCOMPRESSED] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [SNAPPY] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [GZIP] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [LZO] | [1.0.0] | [1.0.0][tree-1.0.0] | | +| [BROTLI] | [2.4.0] | [2.3.1..2.4.0] | | +| [LZ4 (deprecated)] | [2.4.0] | [2.3.1..2.4.0] | | +| [LZ4_RAW] | [2.9.0] | [2.8.0..2.9.0] | | +| [ZSTD] | [2.4.0] | [2.3.1..2.4.0] | | +| [Modular encryption] | [2.7.0] | [2.6.0..2.7.0] | [Approved 2019-01-16] | > **Note:** Files with an [encrypted footer] use different magic bytes (`PARE` From 829d6cf542aa37e047c8caa323b2f0fbbb53f108 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 16 Jun 2026 17:43:43 -0400 Subject: [PATCH 14/17] revert package json --- package.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/package.json b/package.json index d5e65f5e..91825029 100644 --- a/package.json +++ b/package.json @@ -18,9 +18,9 @@ "homepage": "https://github.com/apache/parquet-site#readme", "devDependencies": { "@mermaid-js/mermaid-cli": "^11.4.2", - "autoprefixer": "^10.5.0", + "autoprefixer": "^10.4.18", "bootstrap": "^5.3.6", - "postcss": "^8.5.15", - "postcss-cli": "^11.0.1" + "postcss": "^8.4.35", + "postcss-cli": "^11.0.0" } } From ae621fee2d5c9b24cd826fd555c5a6f2f0d75732 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 16 Jun 2026 17:44:45 -0400 Subject: [PATCH 15/17] Update content/en/docs/File Format/versions.md Co-authored-by: emkornfield --- content/en/docs/File Format/versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index f5bfda86..1e703703 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -58,7 +58,7 @@ incompatible features. ## Adding new features New features are added by discussion and voting on the [parquet dev mailing list] -(full process [here]). Once approved, a feature is added to the spec and ships in +(full process [contributing guide]). Once approved, a feature is added to the spec and ships in the next parquet-format release. [parquet dev mailing list]: https://lists.apache.org/list.html?dev@parquet.apache.org From 9faf941aeceeb2d52f27f98ce3c8051d04b4eb08 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 16 Jun 2026 17:45:20 -0400 Subject: [PATCH 16/17] fix link --- content/en/docs/File Format/versions.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index 1e703703..f5f3277d 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -58,13 +58,13 @@ incompatible features. ## Adding new features New features are added by discussion and voting on the [parquet dev mailing list] -(full process [contributing guide]). Once approved, a feature is added to the spec and ships in +(full process in the [contributing guide]). Once approved, a feature is added to the spec and ships in the next parquet-format release. [parquet dev mailing list]: https://lists.apache.org/list.html?dev@parquet.apache.org [semantic versioning]: https://semver.org/ [`thrift FileMetadata`]: https://github.com/apache/parquet-format/blob/c42c2cb4ecfccb38153375e24b702a82fd763cc0/src/main/thrift/parquet.thrift#L1365-L1373 -[here]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format +[contributing guide]: https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#additionschanges-to-the-format [note in parquet.thrift]: https://github.com/apache/parquet-format/blob/74001e41f5c5a1856b29be115f9c992cab16a4bf/src/main/thrift/parquet.thrift#L1368-L1373 [closing-out-2.0]: https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2 From 9218b90a7322190b59e92b1035879fde15dc553b Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 23 Jun 2026 17:41:09 -0400 Subject: [PATCH 17/17] Update status of IEEE 754 total order and NaN counts --- content/en/docs/File Format/versions.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/content/en/docs/File Format/versions.md b/content/en/docs/File Format/versions.md index f5f3277d..3f666d22 100644 --- a/content/en/docs/File Format/versions.md +++ b/content/en/docs/File Format/versions.md @@ -124,7 +124,7 @@ new information. | [Size statistics] | [2.10.0] | [2.9.0..2.10.0] | [Approved 2023-11-14] | | [Geospatial statistics] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2025-02-09] | | [Binary protocol extensions] | [2.11.0] | [2.10.0..2.11.0] | [Approved 2024-09-06] | -| [IEEE 754 total order and NaN counts] | not yet released | [#514] | [Approved 2026-05-26] | +| [IEEE 754 total order and NaN counts] | [2.13.0] | [2.12.0..2.13.0] | [Approved 2026-05-26] | | [LogicalType union] | [2.4.0] | [2.3.1..2.4.0] | Supersedes `ConvertedType` enum
deprecated in [2.9.0] | | [STRING (BYTE_ARRAY)] | [1.0.0] | [1.0.0][tree-1.0.0] | | | [ENUM (BYTE_ARRAY)] | [2.0.0] | [1.0.0..2.0.0] | | @@ -178,6 +178,7 @@ new information. [2.7.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.7.0 [2.10.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.10.0 [2.12.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.12.0 +[2.13.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.13.0 [Approved 2019-12-03]: https://lists.apache.org/thread/xs5qt2odm299pxgqb22mty2csc1so5yr [Approved 2024-03-18]: https://lists.apache.org/thread/nlsj0ftxy7y4ov1678rgy5zc7dmogg6q @@ -196,6 +197,7 @@ new information. [2.6.0..2.7.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.6.0...apache-parquet-format-2.7.0 [2.9.0..2.10.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.9.0...apache-parquet-format-2.10.0 [2.11.0..2.12.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.11.0...apache-parquet-format-2.12.0 +[2.12.0..2.13.0]: https://github.com/apache/parquet-format/compare/apache-parquet-format-2.12.0...apache-parquet-format-2.13.0 [2.1.0]: https://github.com/apache/parquet-format/releases/tag/parquet-format-2.1.0 [2.2.0]: https://github.com/apache/parquet-format/releases/tag/apache-parquet-format-2.2.0 @@ -209,7 +211,6 @@ new information. [Variant shredding]: https://github.com/apache/parquet-format/blob/master/VariantShredding.md [Geospatial statistics]: https://github.com/apache/parquet-format/blob/master/Geospatial.md#statistics [Binary protocol extensions]: https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md -[#514]: https://github.com/apache/parquet-format/pull/514 [Approved 2024-09-06]: https://lists.apache.org/thread/x3472kldrq5kjnld9ztj1jozz25f40hg [Approved 2026-05-26]: https://lists.apache.org/thread/h0k0hqo0sojqphnbnrkp8b0gmwdzq9on