From 7d966f06629623ebc7c80f63572afbc492d04810 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Mon, 8 Jun 2026 18:36:59 +0000 Subject: [PATCH 01/16] Add FILE type definitions --- LogicalTypes.md | 67 +++++++++++++++++++++++++++++++++- src/main/thrift/parquet.thrift | 16 ++++++++ 2 files changed, 82 insertions(+), 1 deletion(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 690ae3f56..662bb169f 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -635,7 +635,72 @@ The type has two type parameters: The sort order used for `GEOGRAPHY` is undefined. When writing data, no min/max statistics should be saved for this type and if such non-compliant statistics -are found during reading, they must be ignored. +are found during reading, they must be ignored. + +### FILE + +`FILE` annotates a group that represents a reference to an external file, along with +the minimum metadata required to read it. It is intended for use cases such as +storing file inventories, manifests, and unstructured data references (e.g., images +or audio files stored in object storage). + +The annotated group must contain the following fields, identified by name. Field IDs +may also be used for projection: + +| Field | Type | Required | +|----------|--------|----------| +| `path` | STRING | Yes | +| `size` | INT64 | No | +| `offset` | INT64 | No | +| `etag` | STRING | No | + +#### path + +An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special +encoding (e.g., URI encoding) is applied. This is the only required field. + +#### size + +The length of the content in bytes. Must be zero or a positive integer if provided. +A value of 0 indicates an empty file. + +#### offset + +A byte offset for range reads. If not provided, readers must treat the value as 0. +If provided and non-zero, readers must seek to this offset and read `size` bytes. +If `offset` is provided, `size` must also be provided. + +#### etag + +An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage). +Can be used to detect whether the referenced file has been updated. If the reference +points to a byte range within a file, the eTag applies to the entire file. + +Validation rules for readers and writers: + +* The `path` field is required and must be present. Readers must reject a `FILE`-annotated + group that does not contain `path`. +* If `offset` is present and non-zero, `size` must also be provided. +* Additional metadata about the file (e.g., content type, modification timestamp) should + be stored adjacent to this struct by engines or table formats, not inside it. + +Statistics may be collected for the individual fields of a `FILE`-annotated group +according to the sort order of each field's logical type. + +This is an example `FILE`-annotated group in Parquet: + +``` +optional group my_file (FILE) { + required binary path (STRING); + optional int64 size; + optional int64 offset; + optional binary etag (STRING); +} +``` + +*Compatibility* + +`FILE` has no corresponding `ConvertedType`. ## Nested Types diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index fe259d61b..b4f9a1e08 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -468,6 +468,21 @@ struct GeographyType { 2: optional EdgeInterpolationAlgorithm algorithm; } +/** + * File logical type annotation + * + * Annotates a group that represents a reference to an external file. + * The group must contain the following fields identified by name: + * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg) + * - size (INT64, optional): the length of the content in bytes; must be zero or positive + * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided + * - etag (STRING, optional): eTag from the storage system for staleness detection + * + * See LogicalTypes.md for details. + */ +struct FileType { +} + /** * LogicalType annotations to replace ConvertedType. * @@ -501,6 +516,7 @@ union LogicalType { 16: VariantType VARIANT // no compatible ConvertedType 17: GeometryType GEOMETRY // no compatible ConvertedType 18: GeographyType GEOGRAPHY // no compatible ConvertedType + 19: FileType FILE // no compatible ConvertedType } /** From b6ae0de760dd13918beef4aca60b7026c0585969 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 10 Jun 2026 19:32:50 +0000 Subject: [PATCH 02/16] Address comments --- LogicalTypes.md | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 662bb169f..c44333c23 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -654,29 +654,33 @@ may also be used for projection: | `offset` | INT64 | No | | `etag` | STRING | No | -#### path +#### Fields + +##### path An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special encoding (e.g., URI encoding) is applied. This is the only required field. -#### size +##### size The length of the content in bytes. Must be zero or a positive integer if provided. -A value of 0 indicates an empty file. +A value of 0 indicates an empty file. If not provided, the length of the referenced +content is unknown and the entirety of the content can be read. -#### offset +##### offset -A byte offset for range reads. If not provided, readers must treat the value as 0. -If provided and non-zero, readers must seek to this offset and read `size` bytes. +A byte offset indicating the start of a content slice within the referenced file. +If not provided, readers must treat the value as 0. +If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data. If `offset` is provided, `size` must also be provided. -#### etag +##### etag An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage). Can be used to detect whether the referenced file has been updated. If the reference points to a byte range within a file, the eTag applies to the entire file. -Validation rules for readers and writers: +#### Validation * The `path` field is required and must be present. Readers must reject a `FILE`-annotated group that does not contain `path`. From 0f0534088b81950fa77536a07d0920f33f5d59a9 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Thu, 2 Jul 2026 16:30:46 +0000 Subject: [PATCH 03/16] update spec based on Alkis' proposal --- LogicalTypes.md | 127 ++++++++++++++++++++++++--------- src/main/thrift/parquet.thrift | 24 +++++-- 2 files changed, 110 insertions(+), 41 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index c44333c23..cb9e79926 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -639,53 +639,108 @@ are found during reading, they must be ignored. ### FILE -`FILE` annotates a group that represents a reference to an external file, along with -the minimum metadata required to read it. It is intended for use cases such as -storing file inventories, manifests, and unstructured data references (e.g., images -or audio files stored in object storage). - -The annotated group must contain the following fields, identified by name. Field IDs -may also be used for projection: - -| Field | Type | Required | -|----------|--------|----------| -| `path` | STRING | Yes | -| `size` | INT64 | No | -| `offset` | INT64 | No | -| `etag` | STRING | No | +`FILE` annotates a group that represents a reference to a range of bytes, which may +be stored inline in the value, elsewhere within this file, or in an external file. It +is intended for use cases such as storing file inventories, manifests, and unstructured +data references (e.g., images or audio files stored in object storage). + +The annotated group may contain the following fields, identified by name. Field IDs +may also be used for projection. All fields are optional: + +| Field | Type | Required | +|----------------|------------|----------| +| `path` | STRING | No | +| `offset` | INT64 | No | +| `size` | INT64 | No | +| `content_type` | STRING | No | +| `checksum` | STRING | No | +| `inline` | BYTE_ARRAY | No | + +A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; +`content_type` and `checksum` are metadata describing whatever is resolved. #### Fields ##### path -An opaque path string to the referenced file (e.g., `s3://bucket/file.jpg`). No special -encoding (e.g., URI encoding) is applied. This is the only required field. +An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special +encoding (e.g., URI encoding) is applied. If `path` is absent, the value refers to +this file (a self-reference). + +##### offset + +A byte offset indicating the start of the byte range within the referenced data. +If not provided, readers must treat the value as 0. +If provided and non-zero, readers must seek to this offset to retrieve the referenced data. ##### size -The length of the content in bytes. Must be zero or a positive integer if provided. -A value of 0 indicates an empty file. If not provided, the length of the referenced -content is unknown and the entirety of the content can be read. +The byte length of the referenced data. Must be zero or a positive integer if provided. +A value of 0 indicates empty referenced data. If not provided, the range runs to the end +of the referenced data. -##### offset +##### content_type -A byte offset indicating the start of a content slice within the referenced file. -If not provided, readers must treat the value as 0. -If provided and non-zero, readers must seek to this offset and read `size` bytes to retrieve the referenced data. -If `offset` is provided, `size` must also be provided. +The media type (MIME type) of the resolved bytes (e.g., `image/png`). + +##### checksum + +A self-describing integrity token for the resolved bytes, of the form +`:base64()`. It generalizes the storage-system eTag. The +recognized algorithms are: -##### etag +| Algorithm | Notes | +|-----------|----------------------------------------------------------------| +| `ETAG` | the object-store eTag — equality-only, not recomputable | +| `MD5` | the usual S3/HTTP eTag and Content-MD5 | +| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) | +| `CRC32C` | common in object stores, hardware-accelerated | +| `SHA-256` | e.g. S3 additional checksums | -An eTag value provided by the storage system (e.g., from S3 or Azure Blob Storage). -Can be used to detect whether the referenced file has been updated. If the reference -points to a byte range within a file, the eTag applies to the entire file. +`checksum` applies to the resolved bytes, except for `ETAG`, which is the +object-store eTag for the whole file referenced by `path`. + +##### inline + +The referenced bytes stored inline in the value. If `inline` is set, it supplies the +bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance +only. + +#### Resolution + +A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are +set: + +| `inline` | `path` | `offset` | `size` | Resolves to | +|----------|--------|----------|--------|----------------------------------------------| +| set | – | – | – | the inline bytes | +| – | set | – | – | whole external file at `path` | +| – | set | set | – | external `path`, `[offset, EOF)` | +| – | set | – | set | external `path`, `[0, size)` | +| – | set | set | set | external `path`, `[offset, offset + size)` | +| – | – | set | – | this file, `[offset, EOF)` (self-reference) | +| – | – | – | set | this file, `[0, size)` (self-reference) | +| – | – | set | set | this file, `[offset, offset + size)` (self-reference) | +| – | – | – | – | nothing — invalid | + +A self-reference typically points within the same Parquet file using `offset` and +`size`; the bytes are written between column chunks and are not otherwise referenced by +the footer. A self-reference is the absence of `path`, never an absolute path back to +the current file, so a file containing self-references is renamed or relocated as a +single unit. + +The referenced bytes are compressed with the same `CompressionCodec` as the one +specified for the `inline` column. #### Validation -* The `path` field is required and must be present. Readers must reject a `FILE`-annotated - group that does not contain `path`. -* If `offset` is present and non-zero, `size` must also be provided. -* Additional metadata about the file (e.g., content type, modification timestamp) should +* A value must resolve to some referenced data. If none of `inline`, `path`, `offset`, or + `size` is set, the value does not resolve and is invalid; use column nullability to + represent a null value. +* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the + locator fields as mutually exclusive. +* Field names within a `FILE`-annotated group must not be renamed. +* Additional metadata about the file (e.g., modification timestamp) should be stored adjacent to this struct by engines or table formats, not inside it. Statistics may be collected for the individual fields of a `FILE`-annotated group @@ -695,10 +750,12 @@ This is an example `FILE`-annotated group in Parquet: ``` optional group my_file (FILE) { - required binary path (STRING); - optional int64 size; + optional binary path (STRING); optional int64 offset; - optional binary etag (STRING); + optional int64 size; + optional binary content_type (STRING); + optional binary checksum (STRING); + optional binary inline; } ``` diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index b4f9a1e08..d95b16dcf 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -471,12 +471,24 @@ struct GeographyType { /** * File logical type annotation * - * Annotates a group that represents a reference to an external file. - * The group must contain the following fields identified by name: - * - path (STRING, required): an opaque string path to the file (e.g. s3://bucket/file.jpg) - * - size (INT64, optional): the length of the content in bytes; must be zero or positive - * - offset (INT64, optional): byte offset for range reads; if provided, size must also be provided - * - etag (STRING, optional): eTag from the storage system for staleness detection + * Annotates a group that represents a reference to a file, or to a range of + * bytes that may be stored inline, elsewhere in this file, or in an external + * file. All fields are optional and are identified by name: + * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg); + * if absent, the value refers to this file (self-reference) + * - offset (INT64): start of the byte range; if absent, treated as 0 + * - size (INT64): byte length of the referenced data; if absent, the range runs to + * the end of the referenced data + * - content_type (STRING): media type (MIME) of the resolved bytes + * - checksum (STRING): an algorithm-tagged integrity token for the resolved + * bytes, of the form ":base64()" (generalizes etag) + * - inline (BYTE_ARRAY): the referenced bytes stored inline in the value + * + * A value resolves to bytes determined by inline / path / offset / size; if + * inline is set it supplies the bytes and any locator fields are provenance + * only. A value with none of inline, path, offset, or size set does not + * resolve to any referenced data and is invalid (use column nullability for nulls). + * content_type and checksum are metadata describing whichever bytes resolve. * * See LogicalTypes.md for details. */ From bcbabcc67c3345aec8a79ed36a25f32752df79ff Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Thu, 2 Jul 2026 16:50:14 +0000 Subject: [PATCH 04/16] minor changes --- LogicalTypes.md | 4 ++-- src/main/thrift/parquet.thrift | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index cb9e79926..0cb49da54 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -664,8 +664,8 @@ A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; ##### path An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special -encoding (e.g., URI encoding) is applied. If `path` is absent, the value refers to -this file (a self-reference). +encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent, +the value refers to this file (a self-reference). ##### offset diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index d95b16dcf..05a4d4796 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -481,7 +481,7 @@ struct GeographyType { * the end of the referenced data * - content_type (STRING): media type (MIME) of the resolved bytes * - checksum (STRING): an algorithm-tagged integrity token for the resolved - * bytes, of the form ":base64()" (generalizes etag) + * bytes, of the form ":base64()" * - inline (BYTE_ARRAY): the referenced bytes stored inline in the value * * A value resolves to bytes determined by inline / path / offset / size; if From 117b3d1da4c2811f736e1d09e371f5256433e0b0 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Tue, 7 Jul 2026 22:05:37 +0000 Subject: [PATCH 05/16] address comments --- LogicalTypes.md | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 0cb49da54..a1bcb4434 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -640,44 +640,50 @@ are found during reading, they must be ignored. ### FILE `FILE` annotates a group that represents a reference to a range of bytes, which may -be stored inline in the value, elsewhere within this file, or in an external file. It +be stored inline in the value, elsewhere within the current file, or in an external file. It is intended for use cases such as storing file inventories, manifests, and unstructured data references (e.g., images or audio files stored in object storage). The annotated group may contain the following fields, identified by name. Field IDs -may also be used for projection. All fields are optional: - -| Field | Type | Required | -|----------------|------------|----------| -| `path` | STRING | No | -| `offset` | INT64 | No | -| `size` | INT64 | No | -| `content_type` | STRING | No | -| `checksum` | STRING | No | -| `inline` | BYTE_ARRAY | No | +may also be used for projection. All fields have a field repetition type of `OPTIONAL` +and may be omitted: + +| Field | Type | +|----------------|------------| +| `path` | STRING | +| `offset` | INT64 | +| `size` | INT64 | +| `content_type` | STRING | +| `checksum` | STRING | +| `inline` | BYTE_ARRAY | A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. #### Fields +For the descriptions below, a field is *set* when it is present in the `FILE` group +and its value is non-null (and, for string fields, non-empty). A field is *not set* +when it is absent from the group, or is present but null or empty. + ##### path An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special -encoding (e.g., URI encoding) is applied on top of the user provided data. If `path` is absent, -the value refers to this file (a self-reference). +encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is +not set, the value refers to the current file (a self-reference). ##### offset A byte offset indicating the start of the byte range within the referenced data. -If not provided, readers must treat the value as 0. -If provided and non-zero, readers must seek to this offset to retrieve the referenced data. +If not set, readers must treat the value as 0. +If set and non-zero, readers must seek to this offset to retrieve the referenced data. ##### size -The byte length of the referenced data. Must be zero or a positive integer if provided. -A value of 0 indicates empty referenced data. If not provided, the range runs to the end -of the referenced data. +The byte length of the referenced data. Must be zero or a positive integer if set; a +value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set +or `path` is not set. It may be omitted only for a whole-file external reference (`path` +set, `offset` not set), in which case the range runs to the end of the referenced file. ##### content_type @@ -703,7 +709,7 @@ object-store eTag for the whole file referenced by `path`. ##### inline The referenced bytes stored inline in the value. If `inline` is set, it supplies the -bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance +bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance only. #### Resolution From 4ff444e01ea5b4d088277dc24f2276069f183778 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 8 Jul 2026 01:07:27 +0000 Subject: [PATCH 06/16] save changes --- LogicalTypes.md | 32 ++++++++++++++++++-------------- src/main/thrift/parquet.thrift | 6 ++++-- 2 files changed, 22 insertions(+), 16 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index a1bcb4434..463712f44 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -717,17 +717,19 @@ only. A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are set: -| `inline` | `path` | `offset` | `size` | Resolves to | -|----------|--------|----------|--------|----------------------------------------------| -| set | – | – | – | the inline bytes | -| – | set | – | – | whole external file at `path` | -| – | set | set | – | external `path`, `[offset, EOF)` | -| – | set | – | set | external `path`, `[0, size)` | -| – | set | set | set | external `path`, `[offset, offset + size)` | -| – | – | set | – | this file, `[offset, EOF)` (self-reference) | -| – | – | – | set | this file, `[0, size)` (self-reference) | -| – | – | set | set | this file, `[offset, offset + size)` (self-reference) | -| – | – | – | – | nothing — invalid | +| `inline` | `path` | `offset` | `size` | Resolves to | +|----------|--------|----------|--------|-------------------------------------------------------| +| set | – | – | – | the inline bytes | +| – | set | – | – | whole external file at `path` | +| – | set | – | set | external `path`, `[0, size)` | +| – | set | set | set | external `path`, `[offset, offset + size)` | +| – | – | – | set | current file, `[0, size)` (self-reference) | +| – | – | set | set | current file, `[offset, offset + size)` (self-reference) | +| – | – | – | – | nothing — invalid | + +`size` must be set whenever `offset` is set or `path` is not set, so a self-reference +and any offset-based read always carry an explicit `size`. It may be omitted only for a +whole-file external reference, where the range runs to the end of the referenced file. A self-reference typically points within the same Parquet file using `offset` and `size`; the bytes are written between column chunks and are not otherwise referenced by @@ -735,19 +737,21 @@ the footer. A self-reference is the absence of `path`, never an absolute path ba the current file, so a file containing self-references is renamed or relocated as a single unit. -The referenced bytes are compressed with the same `CompressionCodec` as the one -specified for the `inline` column. +The bytes referenced by a self-reference are compressed with the same `CompressionCodec` +as the one specified for the `inline` column. #### Validation * A value must resolve to some referenced data. If none of `inline`, `path`, `offset`, or `size` is set, the value does not resolve and is invalid; use column nullability to represent a null value. +* `size` must be set whenever `offset` is set or `path` is not set. A value that sets + `offset` without `size`, or is a self-reference (no `path`) without `size`, is invalid. * If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) should - be stored adjacent to this struct by engines or table formats, not inside it. + be stored adjacent to this group by engines or table formats, not inside it. Statistics may be collected for the individual fields of a `FILE`-annotated group according to the sort order of each field's logical type. diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index 05a4d4796..16229eeb3 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -477,8 +477,10 @@ struct GeographyType { * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg); * if absent, the value refers to this file (self-reference) * - offset (INT64): start of the byte range; if absent, treated as 0 - * - size (INT64): byte length of the referenced data; if absent, the range runs to - * the end of the referenced data + * - size (INT64): byte length of the referenced data; required when `offset` is set + * or `path` is absent. May be omitted only for a whole-file external reference + * (`path` set, `offset` absent), in which case the range runs to the end of the + * referenced file * - content_type (STRING): media type (MIME) of the resolved bytes * - checksum (STRING): an algorithm-tagged integrity token for the resolved * bytes, of the form ":base64()" From 278c9c0d5f2edc5a0a9000586b611dd323b7234a Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 8 Jul 2026 01:10:54 +0000 Subject: [PATCH 07/16] address more --- LogicalTypes.md | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 463712f44..1a296c4ec 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -645,8 +645,11 @@ is intended for use cases such as storing file inventories, manifests, and unstr data references (e.g., images or audio files stored in object storage). The annotated group may contain the following fields, identified by name. Field IDs -may also be used for projection. All fields have a field repetition type of `OPTIONAL` -and may be omitted: +may also be used for projection. Every field is optional both in the schema and in the +data: a writer may omit any field from the group definition, and any field that is +present has a field repetition type of `OPTIONAL`. A group need only define the fields +it uses (for example, an inline-only group may define just `inline`, and an external +reference may define just `path`). | Field | Type | |----------------|------------| @@ -733,7 +736,7 @@ whole-file external reference, where the range runs to the end of the referenced A self-reference typically points within the same Parquet file using `offset` and `size`; the bytes are written between column chunks and are not otherwise referenced by -the footer. A self-reference is the absence of `path`, never an absolute path back to +the footer. A self-reference is when `path` is not set, never an absolute path back to the current file, so a file containing self-references is renamed or relocated as a single unit. @@ -750,13 +753,13 @@ as the one specified for the `inline` column. * If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. -* Additional metadata about the file (e.g., modification timestamp) should +* Additional metadata about the file (e.g., modification timestamp) must be stored adjacent to this group by engines or table formats, not inside it. Statistics may be collected for the individual fields of a `FILE`-annotated group according to the sort order of each field's logical type. -This is an example `FILE`-annotated group in Parquet: +This is an example of a `FILE`-annotated group that defines all fields: ``` optional group my_file (FILE) { @@ -769,9 +772,25 @@ optional group my_file (FILE) { } ``` -*Compatibility* +Because every field is optional, a group need only define the fields it uses. A group +whose values are always stored inline may define just `inline`: + +``` +optional group inline_file (FILE) { + optional binary inline; + optional binary content_type (STRING); +} +``` + +A group whose values are always whole external files may define just `path`: + +``` +optional group external_file (FILE) { + optional binary path (STRING); + optional binary content_type (STRING); +} +``` -`FILE` has no corresponding `ConvertedType`. ## Nested Types From 7186bca2c70935b2994b78c657f8277e0b0dc792 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 8 Jul 2026 01:20:43 +0000 Subject: [PATCH 08/16] Save --- LogicalTypes.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 1a296c4ec..72e878898 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -782,12 +782,14 @@ optional group inline_file (FILE) { } ``` -A group whose values are always whole external files may define just `path`: +A group whose values are always whole external files may define just `path` and optionally +`content_type` and `checksum` for validation.: ``` optional group external_file (FILE) { optional binary path (STRING); optional binary content_type (STRING); + optional binary checksum (STRING); } ``` From a60d3121f77fddebd336ed2aef783ae3c88fa565 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 8 Jul 2026 18:03:37 +0000 Subject: [PATCH 09/16] address comments --- LogicalTypes.md | 64 +++++++++++++++++++--------------- src/main/thrift/parquet.thrift | 19 +--------- 2 files changed, 37 insertions(+), 46 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 72e878898..126678a13 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -644,9 +644,9 @@ be stored inline in the value, elsewhere within the current file, or in an exter is intended for use cases such as storing file inventories, manifests, and unstructured data references (e.g., images or audio files stored in object storage). -The annotated group may contain the following fields, identified by name. Field IDs -may also be used for projection. Every field is optional both in the schema and in the -data: a writer may omit any field from the group definition, and any field that is +The annotated group may contain the following fields, identified by name case sensitively. +Field IDs may also be used for projection. Every field is optional both in the schema and +in the data: a writer may omit any field from the group definition, and any field that is present has a field repetition type of `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only group may define just `inline`, and an external reference may define just `path`). @@ -680,13 +680,16 @@ not set, the value refers to the current file (a self-reference). A byte offset indicating the start of the byte range within the referenced data. If not set, readers must treat the value as 0. If set and non-zero, readers must seek to this offset to retrieve the referenced data. +`offset` must be set for a self-reference (`path` not set); it is optional for an +external reference (`path` set). ##### size The byte length of the referenced data. Must be zero or a positive integer if set; a -value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set -or `path` is not set. It may be omitted only for a whole-file external reference (`path` -set, `offset` not set), in which case the range runs to the end of the referenced file. +value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. +It may be omitted only for a whole-file external reference (`path` set, `offset` not set), +in which case the range runs to the end of the referenced file. Because a self-reference +always sets `offset`, it always sets `size` as well. ##### content_type @@ -695,16 +698,16 @@ The media type (MIME type) of the resolved bytes (e.g., `image/png`). ##### checksum A self-describing integrity token for the resolved bytes, of the form -`:base64()`. It generalizes the storage-system eTag. The -recognized algorithms are: +`:base64()`. Readers should ignore unknown +algorithms. The recognized algorithms are: -| Algorithm | Notes | -|-----------|----------------------------------------------------------------| -| `ETAG` | the object-store eTag — equality-only, not recomputable | -| `MD5` | the usual S3/HTTP eTag and Content-MD5 | -| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) | -| `CRC32C` | common in object stores, hardware-accelerated | -| `SHA-256` | e.g. S3 additional checksums | +| Algorithm | Notes | +|-----------|-----------------------------------------------------------------| +| `ETAG` | the object-store eTag — equality-only, not recomputable | +| `MD5` | the usual S3/HTTP eTag and Content-MD5 | +| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) | +| `CRC32C` | common in object stores, hardware-accelerated | +| `SHA-256` | e.g. S3 additional checksums | `checksum` applies to the resolved bytes, except for `ETAG`, which is the object-store eTag for the whole file referenced by `path`. @@ -724,18 +727,21 @@ set: |----------|--------|----------|--------|-------------------------------------------------------| | set | – | – | – | the inline bytes | | – | set | – | – | whole external file at `path` | +| – | set | set | - | external `path`, `[offset, EOF)` | | – | set | – | set | external `path`, `[0, size)` | | – | set | set | set | external `path`, `[offset, offset + size)` | -| – | – | – | set | current file, `[0, size)` (self-reference) | -| – | – | set | set | current file, `[offset, offset + size)` (self-reference) | +| – | - | set | - | invalid | +| – | - | - | set | invalid | +| – | – | set | set | this file, `[offset, offset + size)` (self-reference) | | – | – | – | – | nothing — invalid | -`size` must be set whenever `offset` is set or `path` is not set, so a self-reference -and any offset-based read always carry an explicit `size`. It may be omitted only for a -whole-file external reference, where the range runs to the end of the referenced file. +`size` must be set whenever `offset` is set, so any offset-based read always carries an +explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also +`size`. `size` may be omitted only for a whole-file external reference, where the range +runs to the end of the referenced file. -A self-reference typically points within the same Parquet file using `offset` and -`size`; the bytes are written between column chunks and are not otherwise referenced by +A self-reference points within the same Parquet file using `offset` and `size` (both +required); the bytes are written between column chunks and are not otherwise referenced by the footer. A self-reference is when `path` is not set, never an absolute path back to the current file, so a file containing self-references is renamed or relocated as a single unit. @@ -745,11 +751,13 @@ as the one specified for the `inline` column. #### Validation -* A value must resolve to some referenced data. If none of `inline`, `path`, `offset`, or - `size` is set, the value does not resolve and is invalid; use column nullability to - represent a null value. -* `size` must be set whenever `offset` is set or `path` is not set. A value that sets - `offset` without `size`, or is a self-reference (no `path`) without `size`, is invalid. +* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or + `offset` is set; if none of them are set, the value does not resolve and is invalid, even + if `size` is set. Use column nullability to represent a null value. +* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor + `offset` set (and not `inline`) does not resolve and is invalid. +* `size` must be set whenever `offset` is set. A value that sets `offset` without `size` + is invalid. Because a self-reference must set `offset`, it must also set `size`. * If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. @@ -783,7 +791,7 @@ optional group inline_file (FILE) { ``` A group whose values are always whole external files may define just `path` and optionally -`content_type` and `checksum` for validation.: +`content_type` and `checksum` for validation: ``` optional group external_file (FILE) { diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index 16229eeb3..ff9f4213c 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -473,24 +473,7 @@ struct GeographyType { * * Annotates a group that represents a reference to a file, or to a range of * bytes that may be stored inline, elsewhere in this file, or in an external - * file. All fields are optional and are identified by name: - * - path (STRING): an opaque location string (e.g. s3://bucket/file.jpg); - * if absent, the value refers to this file (self-reference) - * - offset (INT64): start of the byte range; if absent, treated as 0 - * - size (INT64): byte length of the referenced data; required when `offset` is set - * or `path` is absent. May be omitted only for a whole-file external reference - * (`path` set, `offset` absent), in which case the range runs to the end of the - * referenced file - * - content_type (STRING): media type (MIME) of the resolved bytes - * - checksum (STRING): an algorithm-tagged integrity token for the resolved - * bytes, of the form ":base64()" - * - inline (BYTE_ARRAY): the referenced bytes stored inline in the value - * - * A value resolves to bytes determined by inline / path / offset / size; if - * inline is set it supplies the bytes and any locator fields are provenance - * only. A value with none of inline, path, offset, or size set does not - * resolve to any referenced data and is invalid (use column nullability for nulls). - * content_type and checksum are metadata describing whichever bytes resolve. + * file. * * See LogicalTypes.md for details. */ From f03610e225e1568943d342505e4e77e3b8d18567 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 8 Jul 2026 18:18:17 +0000 Subject: [PATCH 10/16] address final comments --- LogicalTypes.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 126678a13..edbccc004 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -644,12 +644,12 @@ be stored inline in the value, elsewhere within the current file, or in an exter is intended for use cases such as storing file inventories, manifests, and unstructured data references (e.g., images or audio files stored in object storage). -The annotated group may contain the following fields, identified by name case sensitively. -Field IDs may also be used for projection. Every field is optional both in the schema and -in the data: a writer may omit any field from the group definition, and any field that is -present has a field repetition type of `OPTIONAL`. A group need only define the fields -it uses (for example, an inline-only group may define just `inline`, and an external -reference may define just `path`). +The annotated group may contain the following fields, identified by name case sensitively, +not by field order. Field IDs, if they exist, may also be used for projection. Every field +is optional both in the schema and in the data: a writer may omit any field from the group +definition, and any field that is present has a field repetition type of `OPTIONAL`. +A group need only define the fields it uses (for example, an inline-only group may define +just `inline`, and an external reference may define just `path`). | Field | Type | |----------------|------------| From 8a2ae513cc88d1f8615d1fd2d47bada149879ac9 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Fri, 10 Jul 2026 15:52:23 +0000 Subject: [PATCH 11/16] address comments, update path spec to URI --- LogicalTypes.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index edbccc004..de37556c4 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -671,9 +671,10 @@ when it is absent from the group, or is present but null or empty. ##### path -An opaque path string to an external file (e.g., `s3://bucket/file.jpg`). No special -encoding (e.g., URI encoding) is applied on top of the user-provided data. If `path` is -not set, the value refers to the current file (a self-reference). +A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`). +The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top +of the user-provided data. If `path` is not set, the value refers to the current file +(a self-reference). ##### offset @@ -681,7 +682,7 @@ A byte offset indicating the start of the byte range within the referenced data. If not set, readers must treat the value as 0. If set and non-zero, readers must seek to this offset to retrieve the referenced data. `offset` must be set for a self-reference (`path` not set); it is optional for an -external reference (`path` set). +external reference (`path` set). `offset` must not be < 0. ##### size @@ -727,7 +728,7 @@ set: |----------|--------|----------|--------|-------------------------------------------------------| | set | – | – | – | the inline bytes | | – | set | – | – | whole external file at `path` | -| – | set | set | - | external `path`, `[offset, EOF)` | +| – | set | set | - | invalid | | – | set | – | set | external `path`, `[0, size)` | | – | set | set | set | external `path`, `[offset, offset + size)` | | – | - | set | - | invalid | From 90147dcd4c421ce45bead55cc125d74395927823 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Fri, 17 Jul 2026 21:46:13 +0000 Subject: [PATCH 12/16] address comments --- LogicalTypes.md | 52 +++++++++++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 21 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index de37556c4..d279bd079 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -666,9 +666,11 @@ A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; #### Fields For the descriptions below, a field is *set* when it is present in the `FILE` group -and its value is non-null (and, for string fields, non-empty). A field is *not set* +and its value is non-null (and, for string fields, non-empty[1]). A field is *not set* when it is absent from the group, or is present but null or empty. +[1] Implementations are not expected to treat empty strings as null + ##### path A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`). @@ -694,21 +696,30 @@ always sets `offset`, it always sets `size` as well. ##### content_type -The media type (MIME type) of the resolved bytes (e.g., `image/png`). +The media type (MIME type), as defined by RFC 2046, of the resolved bytes (e.g., `image/png`). +When not set, the type can be assumed as `application/octet-stream`. ##### checksum A self-describing integrity token for the resolved bytes, of the form -`:base64()`. Readers should ignore unknown -algorithms. The recognized algorithms are: +`:`, where `` is encoded according to the `Encoding` +column below. Readers should ignore unknown algorithms. The recognized algorithms +are: + +| Algorithm | Encoding | Notes | +|-----------|---------------|----------------------------------------------------------| +| `ETAG` | opaque | the object-store eTag, not recomputable | +| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters | +| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters | +| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters | +| `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters | + +`` encodings are: -| Algorithm | Notes | -|-----------|-----------------------------------------------------------------| -| `ETAG` | the object-store eTag — equality-only, not recomputable | -| `MD5` | the usual S3/HTTP eTag and Content-MD5 | -| `CRC32` | Parquet's page-checksum algorithm (gzip/zlib) | -| `CRC32C` | common in object stores, hardware-accelerated | -| `SHA-256` | e.g. S3 additional checksums | +* `lowercase hex`: the digest bytes rendered as lowercase hexadecimal, two + characters per byte and no separators (e.g. `MD5:d41d8cd98f00b204e9800998ecf8427e`). +* `opaque`: the token supplied verbatim by the object store, used only for + equality comparison and not otherwise interpreted. `checksum` applies to the resolved bytes, except for `ETAG`, which is the object-store eTag for the whole file referenced by `path`. @@ -742,31 +753,29 @@ explicit `size`. A self-reference (`path` not set) must set `offset`, and theref runs to the end of the referenced file. A self-reference points within the same Parquet file using `offset` and `size` (both -required); the bytes are written between column chunks and are not otherwise referenced by -the footer. A self-reference is when `path` is not set, never an absolute path back to -the current file, so a file containing self-references is renamed or relocated as a -single unit. +required). A self-reference is when `path` is not set. A file containing self-references +can be renamed or relocated as a single unit. -The bytes referenced by a self-reference are compressed with the same `CompressionCodec` -as the one specified for the `inline` column. +The bytes referenced by a self-reference use the `CompressionCodec` +defined by the `inline` column chunk's `ColumnMetadata`. #### Validation * A value must resolve to some referenced data. It resolves only if `inline`, `path`, or `offset` is set; if none of them are set, the value does not resolve and is invalid, even - if `size` is set. Use column nullability to represent a null value. + if `size` is set. * A self-reference (`path` not set) must set `offset`. A value with neither `path` nor `offset` set (and not `inline`) does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` is invalid. Because a self-reference must set `offset`, it must also set `size`. -* If `inline` is set, it supplies the bytes; producers may instead treat `inline` and the +* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) must be stored adjacent to this group by engines or table formats, not inside it. Statistics may be collected for the individual fields of a `FILE`-annotated group -according to the sort order of each field's logical type. +according to the sort order defined in each field's logical type. This is an example of a `FILE`-annotated group that defines all fields: @@ -782,7 +791,8 @@ optional group my_file (FILE) { ``` Because every field is optional, a group need only define the fields it uses. A group -whose values are always stored inline may define just `inline`: +whose values are always stored inline may define just `inline` and optionally `content_type` +as additional metadata: ``` optional group inline_file (FILE) { From 989099b1dca267d160b753d7a3ecb51dc2c39984 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 22 Jul 2026 17:25:28 +0000 Subject: [PATCH 13/16] Address comments --- LogicalTypes.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index d279bd079..d089bd6ac 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -709,9 +709,9 @@ are: | Algorithm | Encoding | Notes | |-----------|---------------|----------------------------------------------------------| | `ETAG` | opaque | the object-store eTag, not recomputable | -| `MD5` | lowercase hex | as defined in RFC 6151 represented as 32 hex characters | -| `CRC32` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters | -| `CRC32C` | lowercase hex | as defined in RFC 9260, represented as 8 hex characters | +| `MD5` | lowercase hex | as defined in RFC 1321 represented as 32 hex characters | +| `CRC32` | lowercase hex | as defined in RFC 2083, represented as 8 hex characters | +| `CRC32C` | lowercase hex | as defined in RFC 3385, represented as 8 hex characters | | `SHA-256` | lowercase hex | as defined in RFC 6234, represented as 64 hex characters | `` encodings are: @@ -737,15 +737,15 @@ set: | `inline` | `path` | `offset` | `size` | Resolves to | |----------|--------|----------|--------|-------------------------------------------------------| -| set | – | – | – | the inline bytes | -| – | set | – | – | whole external file at `path` | -| – | set | set | - | invalid | -| – | set | – | set | external `path`, `[0, size)` | -| – | set | set | set | external `path`, `[offset, offset + size)` | -| – | - | set | - | invalid | -| – | - | - | set | invalid | -| – | – | set | set | this file, `[offset, offset + size)` (self-reference) | -| – | – | – | – | nothing — invalid | +| set | - | - | - | the inline bytes | +| - | set | - | - | whole external file at `path` | +| - | set | set | - | invalid | +| - | set | - | set | external `path`, `[0, size)` | +| - | set | set | set | external `path`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | +| - | - | set | set | this file, `[offset, offset + size)` (self-reference) | +| - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also @@ -756,9 +756,6 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `path` is not set. A file containing self-references can be renamed or relocated as a single unit. -The bytes referenced by a self-reference use the `CompressionCodec` -defined by the `inline` column chunk's `ColumnMetadata`. - #### Validation * A value must resolve to some referenced data. It resolves only if `inline`, `path`, or @@ -768,7 +765,7 @@ defined by the `inline` column chunk's `ColumnMetadata`. `offset` set (and not `inline`) does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` is invalid. Because a self-reference must set `offset`, it must also set `size`. -* If `inline` is set, it supplies the bytes readers; producers may treat `inline` and the +* If `inline` is set, it supplies the bytes for readers; producers may treat `inline` and the locator fields as mutually exclusive. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) must From 92228a13f1de72c8c979644f42d2405c5871fa5e Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Wed, 22 Jul 2026 17:32:18 +0000 Subject: [PATCH 14/16] added invalid read behavior --- LogicalTypes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/LogicalTypes.md b/LogicalTypes.md index d089bd6ac..4c276c03c 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -770,6 +770,8 @@ can be renamed or relocated as a single unit. * Field names within a `FILE`-annotated group must not be renamed. * Additional metadata about the file (e.g., modification timestamp) must be stored adjacent to this group by engines or table formats, not inside it. +* If a reader comes across an invalid file reference, the reader may return a `null` file reference + for that row. Statistics may be collected for the individual fields of a `FILE`-annotated group according to the sort order defined in each field's logical type. From 4223744417a8690cf9a5c702f2cbd24a4759ec93 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Thu, 23 Jul 2026 15:38:42 +0000 Subject: [PATCH 15/16] rename path to uri --- LogicalTypes.md | 58 ++++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 29 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 4c276c03c..e6d7177af 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -649,18 +649,18 @@ not by field order. Field IDs, if they exist, may also be used for projection. E is optional both in the schema and in the data: a writer may omit any field from the group definition, and any field that is present has a field repetition type of `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only group may define -just `inline`, and an external reference may define just `path`). +just `inline`, and an external reference may define just `uri`). | Field | Type | |----------------|------------| -| `path` | STRING | +| `uri` | STRING | | `offset` | INT64 | | `size` | INT64 | | `content_type` | STRING | | `checksum` | STRING | | `inline` | BYTE_ARRAY | -A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; +A value resolves to bytes determined by `inline` / `uri` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. #### Fields @@ -671,11 +671,11 @@ when it is absent from the group, or is present but null or empty. [1] Implementations are not expected to treat empty strings as null -##### path +##### uri A URI-reference as defined by RFC 3986, encoded as a Parquet STRING (e.g., `s3://bucket/file.jpg`). The URI may be absolute or relative. No additional encoding (e.g., URI encoding) is applied on top -of the user-provided data. If `path` is not set, the value refers to the current file +of the user-provided data. If `uri` is not set, the value refers to the current file (a self-reference). ##### offset @@ -683,14 +683,14 @@ of the user-provided data. If `path` is not set, the value refers to the current A byte offset indicating the start of the byte range within the referenced data. If not set, readers must treat the value as 0. If set and non-zero, readers must seek to this offset to retrieve the referenced data. -`offset` must be set for a self-reference (`path` not set); it is optional for an -external reference (`path` set). `offset` must not be < 0. +`offset` must be set for a self-reference (`uri` not set); it is optional for an +external reference (`uri` set). `offset` must not be < 0. ##### size The byte length of the referenced data. Must be zero or a positive integer if set; a value of 0 indicates empty referenced data. `size` must be set whenever `offset` is set. -It may be omitted only for a whole-file external reference (`path` set, `offset` not set), +It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), in which case the range runs to the end of the referenced file. Because a self-reference always sets `offset`, it always sets `size` as well. @@ -722,46 +722,46 @@ are: equality comparison and not otherwise interpreted. `checksum` applies to the resolved bytes, except for `ETAG`, which is the -object-store eTag for the whole file referenced by `path`. +object-store eTag for the whole file referenced by `uri`. ##### inline The referenced bytes stored inline in the value. If `inline` is set, it supplies the -bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance +bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance only. #### Resolution -A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are +A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are set: -| `inline` | `path` | `offset` | `size` | Resolves to | -|----------|--------|----------|--------|-------------------------------------------------------| -| set | - | - | - | the inline bytes | -| - | set | - | - | whole external file at `path` | -| - | set | set | - | invalid | -| - | set | - | set | external `path`, `[0, size)` | -| - | set | set | set | external `path`, `[offset, offset + size)` | -| - | - | set | - | invalid | -| - | - | - | set | invalid | -| - | - | set | set | this file, `[offset, offset + size)` (self-reference) | -| - | - | - | - | nothing - invalid | +| `inline` | `uri` | `offset` | `size` | Resolves to | +|----------|-------|----------|--------|-------------------------------------------------------| +| set | - | - | - | the inline bytes | +| - | set | - | - | whole external file at `uri` | +| - | set | set | - | invalid | +| - | set | - | set | external `uri`, `[0, size)` | +| - | set | set | set | external `uri`, `[offset, offset + size)` | +| - | - | set | - | invalid | +| - | - | - | set | invalid | +| - | - | set | set | this file, `[offset, offset + size)` (self-reference) | +| - | - | - | - | nothing - invalid | `size` must be set whenever `offset` is set, so any offset-based read always carries an -explicit `size`. A self-reference (`path` not set) must set `offset`, and therefore also +explicit `size`. A self-reference (`uri` not set) must set `offset`, and therefore also `size`. `size` may be omitted only for a whole-file external reference, where the range runs to the end of the referenced file. A self-reference points within the same Parquet file using `offset` and `size` (both -required). A self-reference is when `path` is not set. A file containing self-references +required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. #### Validation -* A value must resolve to some referenced data. It resolves only if `inline`, `path`, or +* A value must resolve to some referenced data. It resolves only if `inline`, `uri`, or `offset` is set; if none of them are set, the value does not resolve and is invalid, even if `size` is set. -* A self-reference (`path` not set) must set `offset`. A value with neither `path` nor +* A self-reference (`uri` not set) must set `offset`. A value with neither `uri` nor `offset` set (and not `inline`) does not resolve and is invalid. * `size` must be set whenever `offset` is set. A value that sets `offset` without `size` is invalid. Because a self-reference must set `offset`, it must also set `size`. @@ -780,7 +780,7 @@ This is an example of a `FILE`-annotated group that defines all fields: ``` optional group my_file (FILE) { - optional binary path (STRING); + optional binary uri (STRING); optional int64 offset; optional int64 size; optional binary content_type (STRING); @@ -800,12 +800,12 @@ optional group inline_file (FILE) { } ``` -A group whose values are always whole external files may define just `path` and optionally +A group whose values are always whole external files may define just `uri` and optionally `content_type` and `checksum` for validation: ``` optional group external_file (FILE) { - optional binary path (STRING); + optional binary uri (STRING); optional binary content_type (STRING); optional binary checksum (STRING); } From f241be805fb48f6f8141534b09c9a995e250a3a9 Mon Sep 17 00:00:00 2001 From: Ubuntu Date: Fri, 24 Jul 2026 02:51:57 +0000 Subject: [PATCH 16/16] address Rok's comments --- LogicalTypes.md | 5 +++++ src/main/thrift/parquet.thrift | 1 + 2 files changed, 6 insertions(+) diff --git a/LogicalTypes.md b/LogicalTypes.md index e6d7177af..fae874c0d 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -756,6 +756,11 @@ A self-reference points within the same Parquet file using `offset` and `size` ( required). A self-reference is when `uri` is not set. A file containing self-references can be renamed or relocated as a single unit. +Parquet files containing self-references must not use Parquet modular encryption. +Self-referenced byte ranges are not Parquet encryption modules and therefore cannot +be encrypted or authenticated independently. Encryption of external files referenced +by `uri` is outside the scope of the Parquet format. + #### Validation * A value must resolve to some referenced data. It resolves only if `inline`, `uri`, or diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index ff9f4213c..6698e9197 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -1116,6 +1116,7 @@ union ColumnOrder { * VARIANT - undefined * GEOMETRY - undefined * GEOGRAPHY - undefined + * FILE - undefined * * In the absence of logical types, the sort order is determined by the physical type: * BOOLEAN - false, true