From 0a49081b33355f71c14d73da41cb02f20dd4dad8 Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Mon, 6 Jul 2026 23:43:24 +0000 Subject: [PATCH 01/10] [Protocol RFC] Add File data type Add a protocol RFC for a new `file` data type that stores a reference to a range of bytes located inline, elsewhere in the same data file, or in an external file. Aligned with the Parquet FILE logical type proposed in apache/parquet-format#585. Introduces the `fileType` table feature (Reader v3 / Writer v7), the `file` primitive type name, Parquet physical encoding, per-leaf statistics, and compatibility notes with other Delta features. See #7147 Co-authored-by: Isaac --- protocol_rfcs/README.md | 1 + protocol_rfcs/file-type.md | 183 +++++++++++++++++++++++++++++++++++++ 2 files changed, 184 insertions(+) create mode 100644 protocol_rfcs/file-type.md diff --git a/protocol_rfcs/README.md b/protocol_rfcs/README.md index 2c6bb51ec7d..2e9b44b03a6 100644 --- a/protocol_rfcs/README.md +++ b/protocol_rfcs/README.md @@ -23,6 +23,7 @@ Here is the history of all the RFCs propose/accepted/rejected since Feb 6, 2024, | 2025-03-13 | [checkpoint-protection.md](https://github.com/delta-io/delta/blob/master/protocol_rfcs/checkpoint-protection.md) | https://github.com/delta-io/delta/issues/4152 | Checkpoint Protection | | 2025-03-18 | [iceberg-writer-compat-v1.md](https://github.com/delta-io/delta/blob/master/protocol_rfcs/iceberg-writer-compat-v1.md) | https://github.com/delta-io/delta/issues/4284 | IcebergWriterCompatV1 | | 2025-11-20 | [materialize-partition-columns.md](https://github.com/delta-io/delta/blob/master/protocol_rfcs/materialize-partition-columns.md) | https://github.com/delta-io/delta/issues/5555 | Materialize Partition Columns | +| 2026-07-06 | [file-type.md](https://github.com/delta-io/delta/blob/master/protocol_rfcs/file-type.md) | https://github.com/delta-io/delta/issues/7147 | File Data Type | ### Accepted RFCs diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md new file mode 100644 index 00000000000..53b0128ded5 --- /dev/null +++ b/protocol_rfcs/file-type.md @@ -0,0 +1,183 @@ +# File Data Type +**Associated Github issue for discussions: https://github.com/delta-io/delta/issues/7147** + +This protocol change adds support for the `file` data type. +The `file` data type stores a reference to a range of bytes that may be located inline in the value, elsewhere within the same data file, or in an external file. +It is intended for use cases such as file inventories, manifests, and unstructured-data references (for example, images or audio stored in object storage), which are increasingly common with AI/ML workloads. + +The `file` data type is the Delta mapping of the Parquet `FILE` logical type proposed in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). This RFC is aligned with that proposal: the physical Parquet representation, the field set, and the byte-resolution rules defined here match the Parquet `FILE` type so that a Delta `file` column round-trips through Parquet without loss. + +-------- + +> ***New Section after the [Clustered Table](#clustered-table) section*** + +# File Data Type + +This feature enables support for the `file` data type, which stores a reference to a range of bytes. +A `file` value resolves to bytes that are located in one of three ways: +- **inline** — the bytes are stored directly in the value, +- **self-reference** — the bytes are stored elsewhere within the same data file, or +- **external** — the bytes are stored in a separate file at a given path. + +The schema serialization method is described in [Schema Serialization Format](#schema-serialization-format), and the physical encoding is described in [File data in Parquet](#file-data-in-parquet). + +To support this feature: +- The table must be on Reader Version 3 and Writer Version 7. +- The feature `fileType` must exist in the table `protocol`'s `readerFeatures` and `writerFeatures`. + +## Example JSON-Encoded Delta Table Schema with File types + +``` +{ + "type" : "struct", + "fields" : [ { + "name" : "profile_image", + "type" : "file", + "nullable" : true, + "metadata" : { } + }, { + "name" : "attachments", + "type" : { + "type" : "array", + "elementType" : { + "type" : "file" + }, + "containsNull" : false + }, + "nullable" : false, + "metadata" : { } + } ] +} +``` + +## File data in Parquet + +The `file` data type is represented in Parquet as a group annotated with the Parquet `FILE` logical type, as specified in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). +The group contains the following fields, identified by name. Field IDs may also be used for projection. All fields are optional: + +Struct field name | Parquet primitive type | Description +-|-|- +path | binary (`STRING`) | An opaque location string for an external file (for example, `s3://bucket/file.jpg`). No encoding (such as URI encoding) is applied on top of the user-provided value. If `path` is absent, the value refers to this file (a self-reference). +offset | int64 | The start of the byte range within the referenced data. If absent, readers must treat it as 0. +size | int64 | The byte length of the referenced data. Must be zero or a positive integer if provided; 0 indicates empty referenced data. If absent, the range runs to the end of the referenced data. +content_type | binary (`STRING`) | The media type (MIME type) of the resolved bytes, for example `image/png`. +checksum | binary (`STRING`) | A self-describing integrity token for the resolved bytes, of the form `:base64()` (see [Checksum](#checksum)). +inline | binary | The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance only. + +A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. The resolution rules are given in [Byte Resolution](#byte-resolution). + +The annotated Parquet group is, for example: + +``` +optional group profile_image (FILE) { + optional binary path (STRING); + optional int64 offset; + optional int64 size; + optional binary content_type (STRING); + optional binary checksum (STRING); + optional binary inline; +} +``` + +### Checksum + +The `checksum` field is a self-describing token of the form `:base64()`. It generalizes the storage-system eTag. The recognized algorithms are: + +Algorithm | Notes +-|- +`ETAG` | The object-store eTag — equality-only, not recomputable. +`MD5` | The usual S3/HTTP eTag and Content-MD5. +`CRC32` | Parquet's page-checksum algorithm (gzip/zlib). +`CRC32C` | Common in object stores, hardware-accelerated. +`SHA-256` | For example, S3 additional checksums. + +`checksum` applies to the resolved bytes, except for `ETAG`, which is the object-store eTag for the whole file referenced by `path`. + +### Byte Resolution + +A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are set: + +`inline` | `path` | `offset` | `size` | Resolves to +:-:|:-:|:-:|:-:|- +set | – | – | – | The `inline` bytes. +– | set | – | – | The whole external file at `path`. +– | set | set | – | External `path`, `[offset, EOF)`. +– | set | – | set | External `path`, `[0, size)`. +– | set | set | set | External `path`, `[offset, offset + size)`. +– | – | set | – | This file, `[offset, EOF)` (self-reference). +– | – | – | set | This file, `[0, size)` (self-reference). +– | – | set | set | This file, `[offset, offset + size)` (self-reference). +– | – | – | – | Nothing — invalid (use column nullability for a null value). + +A self-reference typically points within the same Parquet data file using `offset` and `size`; the bytes are written between column chunks and are not otherwise referenced by the footer. A self-reference is the *absence* of `path`, never an absolute path back to the current file, so a data file containing self-references is renamed or relocated as a single unit. + +The referenced bytes (both `inline` and self-reference regions) are compressed with the same `CompressionCodec` as the one configured for the `inline` column; `size` is therefore the length of the compressed region on disk. External referents (`path` set) are opaque to Parquet and stored as-is. + +## Writer Requirements for File Data Type + +When File type is supported (`writerFeatures` field of a table's `protocol` action contains `fileType`), writers: +- must write a column of type `file` to Parquet as a group annotated with the Parquet `FILE` logical type, containing only fields drawn from the set `path`, `offset`, `size`, `content_type`, `checksum`, and `inline`, with the Parquet primitive types described in [File data in Parquet](#file-data-in-parquet). +- must not rename the fields within a `FILE`-annotated group; the field names above are normative. +- must write every value so that it resolves to a referent according to [Byte Resolution](#byte-resolution); a value with none of `inline`, `path`, `offset`, or `size` set is invalid and must be represented as a column null instead. +- must, when writing a `checksum`, use the `:base64()` form with one of the recognized algorithms in [Checksum](#checksum). +- must store additional metadata about a file (for example, a modification timestamp) adjacent to the `file` column, not inside the `FILE`-annotated group. + +## Reader Requirements for File Data Type + +When File type is supported (`readerFeatures` field of a table's `protocol` action contains `fileType`), readers: +- must recognize and tolerate a `file` data type in a Delta schema. +- must use the correct physical schema (a Parquet `FILE`-annotated group with the optional fields described in [File data in Parquet](#file-data-in-parquet)) when reading a `file` data type from a file. +- must resolve each value to bytes according to [Byte Resolution](#byte-resolution), including the self-reference case (locating the bytes within the same data file when `path` is absent). +- must make the column available to the engine: + - [Recommended] Expose and interpret the group as a single `file` value, resolving inline, self-reference, and external bytes on access. + - [Alternate] Expose the raw physical group (the set of present fields), for example if the engine does not natively support the `file` type. + +## Compatibility with other Delta Features + +Feature | Support for File Data Type +-|- +Partition Columns | **Supported:** A `file` column is allowed to be a non-partitioned column of a partitioned table.
**Unsupported:** A `file` value is a group and cannot be serialized to a partition-value string, so a `file` column cannot be a partition column. +Clustered Tables | **Supported:** A `file` column is allowed to be a non-clustering column of a clustered table.
**Unsupported:** A `file` value is a group and is not a comparable data type as a whole, so a `file` column cannot be a clustering column. +Delta Column Statistics | **Supported:** A `file` column supports the `nullCount` statistic, and `minValues` / `maxValues` on its comparable leaf fields. See [Statistics for File Columns](#statistics-for-file-columns).
**Unsupported:** The `file` column as a whole is not a comparable data type, and the `inline` field does not support `minValues` / `maxValues`. +Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression, as long as the `file` type is not the result type of the generated column expression.
**Unsupported:** The `file` data type is not allowed to be the result type of a generated column expression. +Delta CHECK Constraints | **Supported:** A `file` column is allowed to be used for a CHECK constraint expression. +Default Column Values | **Supported:** A `file` column is allowed to have a default column value. +Change Data Feed | **Supported:** A table using the `file` data type is allowed to enable the Delta Change Data Feed. + +## Statistics for File Columns + +A `file` value is physically a group of leaf fields (see [File data in Parquet](#file-data-in-parquet)), and Delta's [Per-file Statistics](#per-file-statistics) are already encoded mirroring the schema of the data, descending into nested fields. Statistics for a `file` column follow that same per-leaf model, with one exception for the `inline` field: + +- The `nullCount` statistic is collected for the `file` column itself (whether the whole `file` value is null), following the standard nested-field statistics encoding. +- `minValues` and `maxValues` are collected per leaf field, for the comparable leaf fields only: `path` (STRING), `offset` (INT64), `size` (INT64), `content_type` (STRING), and `checksum` (STRING). These follow the standard rules for their respective types (for example, STRING leaves such as `path` are truncated to a fixed prefix length, as with any string column). +- `minValues` and `maxValues` are **not** collected for the `inline` field, because it is binary content for which min/max provides no data-skipping value and may be large. + +Collecting `minValues` / `maxValues` on `path` in particular enables data skipping on file-inventory and manifest tables that filter by path (for example, an object-store prefix). + +The set of columns for which statistics are collected is otherwise governed by the table's existing statistics configuration (for example, the number of indexed columns). + +-------- + +> ***New Sub-Section after the [Variant Type](#variant-type) sub-section within the [Schema Serialization Format](#schema-serialization-format) section*** + +### File Type + +File data uses the Delta type name `file` for Delta schema serialization. + +Field Name | Description +-|- +type | Always the string "file" + +-------- + +> ***Update the [Primitive Types](#primitive-types) table in the [Schema Serialization Format](#schema-serialization-format) section*** + +Add the following row to the Primitive Types table: + +> file | A reference to a range of bytes located inline, elsewhere in the same data file, or in an external file. When stored in a Parquet file it is a group annotated with the Parquet `FILE` logical type. To use this type, a table must support the feature `fileType`. See section [File Data Type](#file-data-type). + +-------- + +> ***Add a row to the [Valid Feature Names in Table Features](#valid-feature-names-in-table-features) table*** + +> [File Data Type](#file-data-type) | `fileType` | Readers and writers From 7f1eb8c767e3d8b5b01f1bf3522e27e8bed5232e Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Tue, 7 Jul 2026 23:10:47 +0000 Subject: [PATCH 02/10] [Protocol RFC] Address review: clarify self-reference, open generated-columns question - Expand the self-reference bullet in the intro with how the bytes are located within the same data file, linking to Byte Resolution. - Reframe the generated-column result-type clause as an open question for discussion rather than a hard restriction inherited from the variant RFC. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 53b0128ded5..8b3d458e649 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -16,8 +16,8 @@ The `file` data type is the Delta mapping of the Parquet `FILE` logical type pro This feature enables support for the `file` data type, which stores a reference to a range of bytes. A `file` value resolves to bytes that are located in one of three ways: - **inline** — the bytes are stored directly in the value, -- **self-reference** — the bytes are stored elsewhere within the same data file, or -- **external** — the bytes are stored in a separate file at a given path. +- **self-reference** — the bytes are stored within the same data file that holds this `file` value, addressed by a byte range (`offset` / `size`) with no `path`. The bytes are written between column chunks and are not otherwise referenced by the Parquet footer, so a reader locates them within the current file rather than opening an external one. See [Byte Resolution](#byte-resolution). +- **external** — the bytes are stored in a separate file at a given `path`. The schema serialization method is described in [Schema Serialization Format](#schema-serialization-format), and the physical encoding is described in [File data in Parquet](#file-data-in-parquet). @@ -139,7 +139,7 @@ Feature | Support for File Data Type Partition Columns | **Supported:** A `file` column is allowed to be a non-partitioned column of a partitioned table.
**Unsupported:** A `file` value is a group and cannot be serialized to a partition-value string, so a `file` column cannot be a partition column. Clustered Tables | **Supported:** A `file` column is allowed to be a non-clustering column of a clustered table.
**Unsupported:** A `file` value is a group and is not a comparable data type as a whole, so a `file` column cannot be a clustering column. Delta Column Statistics | **Supported:** A `file` column supports the `nullCount` statistic, and `minValues` / `maxValues` on its comparable leaf fields. See [Statistics for File Columns](#statistics-for-file-columns).
**Unsupported:** The `file` column as a whole is not a comparable data type, and the `inline` field does not support `minValues` / `maxValues`. -Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression, as long as the `file` type is not the result type of the generated column expression.
**Unsupported:** The `file` data type is not allowed to be the result type of a generated column expression. +Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression.
**Open question:** Whether `file` may be the *result* type of a generated column expression (for example, constructing a `file` reference from other columns) is left open for discussion on the associated issue, and is not specified by this RFC. Delta CHECK Constraints | **Supported:** A `file` column is allowed to be used for a CHECK constraint expression. Default Column Values | **Supported:** A `file` column is allowed to have a default column value. Change Data Feed | **Supported:** A table using the `file` data type is allowed to enable the Delta Change Data Feed. From 244afa3f60b8aa0e1c84d5fcf0893864a879896d Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Tue, 7 Jul 2026 23:34:52 +0000 Subject: [PATCH 03/10] [Protocol RFC] Address review: time travel/CDF caveat and lifecycle non-goals - Add a "Time Travel and Change Data Feed" section clarifying that Delta time- travels and CDFs the reference, not the referenced bytes, which live outside the transaction log and may be overwritten or deleted independently. - Add a "Non-Goals" section scoping out lifecycle/GC of referenced bytes (including any Delta-managed notion and its VACUUM interaction) and access brokering, keeping the type reference-only and aligned with Parquet/Iceberg. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 8b3d458e649..c37beb3e104 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -142,7 +142,7 @@ Delta Column Statistics | **Supported:** A `file` column supports the `nullCount Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression.
**Open question:** Whether `file` may be the *result* type of a generated column expression (for example, constructing a `file` reference from other columns) is left open for discussion on the associated issue, and is not specified by this RFC. Delta CHECK Constraints | **Supported:** A `file` column is allowed to be used for a CHECK constraint expression. Default Column Values | **Supported:** A `file` column is allowed to have a default column value. -Change Data Feed | **Supported:** A table using the `file` data type is allowed to enable the Delta Change Data Feed. +Change Data Feed | **Supported:** A table using the `file` data type is allowed to enable the Delta Change Data Feed. A `file` value is an ordinary column value, so it flows through Change Data Feed and time travel like any other column. See [Time Travel and Change Data Feed](#time-travel-and-change-data-feed) for the distinction between the reference and the referenced bytes. ## Statistics for File Columns @@ -156,6 +156,22 @@ Collecting `minValues` / `maxValues` on `path` in particular enables data skippi The set of columns for which statistics are collected is otherwise governed by the table's existing statistics configuration (for example, the number of indexed columns). +## Time Travel and Change Data Feed + +A `file` value is a reference, and it is stored in the table's data files like any other column value. Delta therefore time-travels and change-data-feeds the **reference**: querying a historical version of the table, or reading `file` columns through the Change Data Feed, returns the reference values exactly as they were written at that version. + +Delta makes **no guarantee about the referenced bytes**, because those bytes live outside the Delta transaction log: + +- The bytes may be overwritten or deleted independently of the table, so dereferencing a reference read from a historical version (via time travel or Change Data Feed) may fail or may return different bytes than when the reference was written. The `checksum` field, when present, allows a reader to detect that the bytes have changed, but does not allow it to recover the original bytes. +- Availability of the referenced bytes is orthogonal to which table version is queried: time travel of the reference does not imply time travel of the bytes. + +## Non-Goals + +The following are out of scope for this RFC: + +- **Lifecycle and garbage collection of referenced bytes.** This RFC defines `file` as a reference only; it does not specify how, or whether, the referenced bytes are created, retained, or reclaimed. In particular, it does not define a notion of Delta-managed referenced bytes or their interaction with `VACUUM`. Referenced bytes are managed out-of-band by the writer or an external system. +- **Access brokering and governance** of the referenced bytes (for example, catalog-vended credentials or signed URLs). + -------- > ***New Sub-Section after the [Variant Type](#variant-type) sub-section within the [Schema Serialization Format](#schema-serialization-format) section*** From 6cba74caf442e1eb4c7e2c30c495a51b7510062f Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Wed, 8 Jul 2026 14:23:37 +0000 Subject: [PATCH 04/10] [Protocol RFC] Add storage mode (managed/external) in column metadata Standardize the representation of a file column's storage mode (MANAGED/EXTERNAL/UNKNOWN) in the __FILE_TYPE_MODE schema-metadata key, a path-keyed map on the nearest ancestor StructField, mirroring the __COLLATIONS representation and DBR's FileType mode (databricks-eng/runtime#213587). This lets the qualifier round-trip through the Delta log without placing it inside the file value (which must round-trip through the fixed Parquet FILE field set). Only the representation is standardized; MANAGED vs EXTERNAL carries no Delta behavior yet, and lifecycle/GC remains a non-goal. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index c37beb3e104..34d89669427 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -165,11 +165,32 @@ Delta makes **no guarantee about the referenced bytes**, because those bytes liv - The bytes may be overwritten or deleted independently of the table, so dereferencing a reference read from a historical version (via time travel or Change Data Feed) may fail or may return different bytes than when the reference was written. The `checksum` field, when present, allows a reader to detect that the bytes have changed, but does not allow it to recover the original bytes. - Availability of the referenced bytes is orthogonal to which table version is queried: time travel of the reference does not imply time travel of the bytes. +## Storage Mode + +A `file` column optionally carries a **storage mode** that records whether the referenced bytes are `managed` (their lifecycle is intended to be governed by the table/engine) or `external` (owned entirely outside the table), or leaves the column unqualified. The storage mode is a property of the *column*, not of an individual value, and it is stored in the column's [schema metadata](#schema-serialization-format) rather than inside the `file` value. This is intentional: the physical `file` value must round-trip through the Parquet `FILE` logical type (and equivalents such as Iceberg's file reference), whose field set is fixed, so a mode stored inside the value would not survive the round-trip. Keeping the mode in schema metadata also lets it propagate through the Delta transaction log unchanged. + +The storage mode is recorded in the `__FILE_TYPE_MODE` key of the metadata of the nearest ancestor [StructField](#struct-field), as a JSON object mapping a field path to one of `MANAGED`, `EXTERNAL`, or `UNKNOWN`. Nested maps and arrays are encoded using the same field-path convention as identifiers in [IcebergCompatV2](#writer-requirements-for-icebergcompatv2) (for example, `arr.element`). This mirrors the representation used for the [Collated String Type](#collated-string-type) (`__COLLATIONS`). A `file` column with no `__FILE_TYPE_MODE` entry is an unqualified `file` reference. + +This RFC standardizes only the **representation** of the storage mode, so that a mode set by one engine propagates through the log and is visible to others. It does **not** assign any behavioral difference to `MANAGED` versus `EXTERNAL` in Delta: byte lifecycle, garbage collection, and access brokering are out of scope (see [Non-Goals](#non-goals)). Engines may attach their own semantics to the mode, but must not assume that reading or writing a mode changes how Delta itself manages the referenced bytes. + +Example schema for `profile_image FILE MANAGED` (irrelevant fields stripped): + +``` +{ + "name" : "profile_image", + "type" : "file", + "nullable" : true, + "metadata" : { + "__FILE_TYPE_MODE" : { "profile_image" : "MANAGED" } + } +} +``` + ## Non-Goals The following are out of scope for this RFC: -- **Lifecycle and garbage collection of referenced bytes.** This RFC defines `file` as a reference only; it does not specify how, or whether, the referenced bytes are created, retained, or reclaimed. In particular, it does not define a notion of Delta-managed referenced bytes or their interaction with `VACUUM`. Referenced bytes are managed out-of-band by the writer or an external system. +- **Lifecycle and garbage collection of referenced bytes.** This RFC defines `file` as a reference only; it does not specify how, or whether, the referenced bytes are created, retained, or reclaimed. In particular, although the [Storage Mode](#storage-mode) may record that a column is `managed`, this RFC does not define any resulting behavior, nor any interaction with `VACUUM`. Referenced bytes are managed out-of-band by the writer or an external system. - **Access brokering and governance** of the referenced bytes (for example, catalog-vended credentials or signed URLs). -------- @@ -194,6 +215,12 @@ Add the following row to the Primitive Types table: -------- +> ***Add a row to the [Column Metadata](#column-metadata) table*** + +> \_\_FILE\_TYPE\_MODE | The [storage mode](#storage-mode) (`MANAGED`, `EXTERNAL`, or `UNKNOWN`) of `file`-typed fields, or combinations of maps and arrays, stored in this field. Encoded as a JSON object mapping field path to mode. See [Storage Mode](#storage-mode) for details. + +-------- + > ***Add a row to the [Valid Feature Names in Table Features](#valid-feature-names-in-table-features) table*** > [File Data Type](#file-data-type) | `fileType` | Readers and writers From 134ceed6fbe3f1081b11c0e47116b9f524495212 Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Thu, 9 Jul 2026 10:47:54 +0000 Subject: [PATCH 05/10] [Protocol RFC] Sync File type with updated Parquet FILE spec (PR #585) Align with the latest apache/parquet-format#585: - size must be set whenever offset is set; a self-reference (no path) must set offset, and therefore size. Drop the now-invalid [offset, EOF) and [0, size) self-reference modes and the external [offset, EOF) mode from the resolution table; add explicit invalid rows. - Define "set" (present, non-null, non-empty for strings) and allow sparse group definitions (a group need only define the fields it uses); add an inline-only example group. - Fields matched case-sensitively by name; field IDs "if they exist". - Readers should ignore unknown checksum algorithms. Follows the consistent prose intent of PR #585; note its resolution table still lists an [offset, EOF) row that contradicts its own validation section (offset requires size) -- to be raised on the Parquet PR. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 39 ++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 34d89669427..19bcd38de71 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -53,20 +53,22 @@ To support this feature: ## File data in Parquet The `file` data type is represented in Parquet as a group annotated with the Parquet `FILE` logical type, as specified in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). -The group contains the following fields, identified by name. Field IDs may also be used for projection. All fields are optional: +The group may contain the following fields, identified by name (matched case-sensitively, not by field order). Field IDs, if they exist, may also be used for projection. Every field is optional both in the schema and in the data: a writer may omit any field from the group definition, and any field that is present has repetition type `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only column may define just `inline`, and a whole-file external reference may define just `path`). + +A field is *set* when it is present in the group and its value is non-null (and, for string fields, non-empty). A field is *not set* when it is absent from the group, or is present but null or empty. Struct field name | Parquet primitive type | Description -|-|- -path | binary (`STRING`) | An opaque location string for an external file (for example, `s3://bucket/file.jpg`). No encoding (such as URI encoding) is applied on top of the user-provided value. If `path` is absent, the value refers to this file (a self-reference). -offset | int64 | The start of the byte range within the referenced data. If absent, readers must treat it as 0. -size | int64 | The byte length of the referenced data. Must be zero or a positive integer if provided; 0 indicates empty referenced data. If absent, the range runs to the end of the referenced data. +path | binary (`STRING`) | An opaque location string for an external file (for example, `s3://bucket/file.jpg`). No encoding (such as URI encoding) is applied on top of the user-provided value. If `path` is not set, the value refers to the current file (a self-reference). +offset | int64 | The start of the byte range within the referenced data. If not set, readers must treat it as 0; if set and non-zero, readers must seek to this offset. `offset` must be set for a self-reference (`path` not set), and is optional for an external reference. +size | int64 | The byte length of the referenced data. Must be zero or a positive integer if set; 0 indicates empty referenced data. `size` must be set whenever `offset` is set. It may be omitted only for a whole-file external reference (`path` set, `offset` not set), in which case the range runs to the end of the referenced file. content_type | binary (`STRING`) | The media type (MIME type) of the resolved bytes, for example `image/png`. checksum | binary (`STRING`) | A self-describing integrity token for the resolved bytes, of the form `:base64()` (see [Checksum](#checksum)). -inline | binary | The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`path`, `offset`, `size`) that are present are provenance only. +inline | binary | The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance only. A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. The resolution rules are given in [Byte Resolution](#byte-resolution). -The annotated Parquet group is, for example: +Because every field is optional, a group need only define the fields it uses. An example group that defines all fields: ``` optional group profile_image (FILE) { @@ -79,9 +81,18 @@ optional group profile_image (FILE) { } ``` +A column whose values are always stored inline may define just `inline` (optionally with `content_type`): + +``` +optional group inline_file (FILE) { + optional binary inline; + optional binary content_type (STRING); +} +``` + ### Checksum -The `checksum` field is a self-describing token of the form `:base64()`. It generalizes the storage-system eTag. The recognized algorithms are: +The `checksum` field is a self-describing token of the form `:base64()`. It generalizes the storage-system eTag. Readers should ignore unknown algorithms. The recognized algorithms are: Algorithm | Notes -|- @@ -95,30 +106,30 @@ Algorithm | Notes ### Byte Resolution -A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are set: +A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are set. `size` must be set whenever `offset` is set, so every offset-based read carries an explicit `size`; `size` may be omitted only for a whole-file external reference. `inline` | `path` | `offset` | `size` | Resolves to :-:|:-:|:-:|:-:|- set | – | – | – | The `inline` bytes. – | set | – | – | The whole external file at `path`. -– | set | set | – | External `path`, `[offset, EOF)`. – | set | – | set | External `path`, `[0, size)`. – | set | set | set | External `path`, `[offset, offset + size)`. -– | – | set | – | This file, `[offset, EOF)` (self-reference). -– | – | – | set | This file, `[0, size)` (self-reference). – | – | set | set | This file, `[offset, offset + size)` (self-reference). +– | set | set | – | Invalid (`offset` set without `size`). +– | – | set | – | Invalid (`offset` set without `size`). +– | – | – | set | Invalid (`size` set without `path` or `offset`). – | – | – | – | Nothing — invalid (use column nullability for a null value). -A self-reference typically points within the same Parquet data file using `offset` and `size`; the bytes are written between column chunks and are not otherwise referenced by the footer. A self-reference is the *absence* of `path`, never an absolute path back to the current file, so a data file containing self-references is renamed or relocated as a single unit. +A self-reference points within the same Parquet data file using `offset` and `size` (both required); the bytes are written between column chunks and are not otherwise referenced by the footer. A self-reference is the *absence* of `path`, never an absolute path back to the current file, so a data file containing self-references is renamed or relocated as a single unit. The referenced bytes (both `inline` and self-reference regions) are compressed with the same `CompressionCodec` as the one configured for the `inline` column; `size` is therefore the length of the compressed region on disk. External referents (`path` set) are opaque to Parquet and stored as-is. ## Writer Requirements for File Data Type When File type is supported (`writerFeatures` field of a table's `protocol` action contains `fileType`), writers: -- must write a column of type `file` to Parquet as a group annotated with the Parquet `FILE` logical type, containing only fields drawn from the set `path`, `offset`, `size`, `content_type`, `checksum`, and `inline`, with the Parquet primitive types described in [File data in Parquet](#file-data-in-parquet). +- must write a column of type `file` to Parquet as a group annotated with the Parquet `FILE` logical type, whose fields are drawn from the set `path`, `offset`, `size`, `content_type`, `checksum`, and `inline`, with the Parquet primitive types described in [File data in Parquet](#file-data-in-parquet). The group need only define the fields it uses; any field it does define must be optional. - must not rename the fields within a `FILE`-annotated group; the field names above are normative. -- must write every value so that it resolves to a referent according to [Byte Resolution](#byte-resolution); a value with none of `inline`, `path`, `offset`, or `size` set is invalid and must be represented as a column null instead. +- must write every value so that it resolves to a referent according to [Byte Resolution](#byte-resolution). In particular, `size` must be set whenever `offset` is set, and a self-reference (`path` not set) must set `offset` (and therefore `size`). A value that does not resolve to any referent is invalid and must be represented as a column null instead. - must, when writing a `checksum`, use the `:base64()` form with one of the recognized algorithms in [Checksum](#checksum). - must store additional metadata about a file (for example, a modification timestamp) adjacent to the `file` column, not inside the `FILE`-annotated group. From 33a213200e91f805c2e8322588d4e30aba9e9cbc Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:31:30 +0000 Subject: [PATCH 06/10] [Protocol RFC] Sync File type with Parquet FILE spec (PR #585 @ 90147dc) Align with the latest apache/parquet-format#585 (commit 90147dc): - path is now a URI-reference per RFC 3986 (absolute or relative), not an opaque location string. Resolves the path/uri debate: name stays `path`, semantics become URI. - checksum form changes from :base64() to : with per-algorithm encoding (lowercase hex for MD5/CRC32/CRC32C/SHA-256, opaque for ETAG); add RFC references and an Encoding column. - content_type: MIME per RFC 2046, defaults to application/octet-stream when unset. - offset must not be negative. - Note that implementations are not expected to treat empty strings as null. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 19bcd38de71..5e20fd2969f 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -55,15 +55,15 @@ To support this feature: The `file` data type is represented in Parquet as a group annotated with the Parquet `FILE` logical type, as specified in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). The group may contain the following fields, identified by name (matched case-sensitively, not by field order). Field IDs, if they exist, may also be used for projection. Every field is optional both in the schema and in the data: a writer may omit any field from the group definition, and any field that is present has repetition type `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only column may define just `inline`, and a whole-file external reference may define just `path`). -A field is *set* when it is present in the group and its value is non-null (and, for string fields, non-empty). A field is *not set* when it is absent from the group, or is present but null or empty. +A field is *set* when it is present in the group and its value is non-null (and, for string fields, non-empty). A field is *not set* when it is absent from the group, or is present but null or empty. (Implementations are not expected to treat empty strings as null.) Struct field name | Parquet primitive type | Description -|-|- -path | binary (`STRING`) | An opaque location string for an external file (for example, `s3://bucket/file.jpg`). No encoding (such as URI encoding) is applied on top of the user-provided value. If `path` is not set, the value refers to the current file (a self-reference). -offset | int64 | The start of the byte range within the referenced data. If not set, readers must treat it as 0; if set and non-zero, readers must seek to this offset. `offset` must be set for a self-reference (`path` not set), and is optional for an external reference. +path | binary (`STRING`) | A URI-reference as defined by [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986), which may be absolute or relative (for example, `s3://bucket/file.jpg`). No additional encoding (such as URI encoding) is applied on top of the user-provided value. If `path` is not set, the value refers to the current file (a self-reference). +offset | int64 | The start of the byte range within the referenced data. Must not be negative. If not set, readers must treat it as 0; if set and non-zero, readers must seek to this offset. `offset` must be set for a self-reference (`path` not set), and is optional for an external reference. size | int64 | The byte length of the referenced data. Must be zero or a positive integer if set; 0 indicates empty referenced data. `size` must be set whenever `offset` is set. It may be omitted only for a whole-file external reference (`path` set, `offset` not set), in which case the range runs to the end of the referenced file. -content_type | binary (`STRING`) | The media type (MIME type) of the resolved bytes, for example `image/png`. -checksum | binary (`STRING`) | A self-describing integrity token for the resolved bytes, of the form `:base64()` (see [Checksum](#checksum)). +content_type | binary (`STRING`) | The media type (MIME type), as defined by [RFC 2046](https://datatracker.ietf.org/doc/html/rfc2046), of the resolved bytes, for example `image/png`. When not set, the type may be assumed to be `application/octet-stream`. +checksum | binary (`STRING`) | A self-describing integrity token for the resolved bytes, of the form `:` (see [Checksum](#checksum)). inline | binary | The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance only. A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. The resolution rules are given in [Byte Resolution](#byte-resolution). @@ -92,15 +92,19 @@ optional group inline_file (FILE) { ### Checksum -The `checksum` field is a self-describing token of the form `:base64()`. It generalizes the storage-system eTag. Readers should ignore unknown algorithms. The recognized algorithms are: +The `checksum` field is a self-describing token of the form `:`, where `` is encoded according to the `Encoding` column below. It generalizes the storage-system eTag. Readers should ignore unknown algorithms. The recognized algorithms are: -Algorithm | Notes --|- -`ETAG` | The object-store eTag — equality-only, not recomputable. -`MD5` | The usual S3/HTTP eTag and Content-MD5. -`CRC32` | Parquet's page-checksum algorithm (gzip/zlib). -`CRC32C` | Common in object stores, hardware-accelerated. -`SHA-256` | For example, S3 additional checksums. +Algorithm | Encoding | Notes +-|-|- +`ETAG` | opaque | The object-store eTag, not recomputable. +`MD5` | lowercase hex | As defined in [RFC 6151](https://datatracker.ietf.org/doc/html/rfc6151), represented as 32 hex characters. +`CRC32` | lowercase hex | As defined in [RFC 3385](https://datatracker.ietf.org/doc/html/rfc3385), represented as 8 hex characters. +`CRC32C` | lowercase hex | As defined in [RFC 9260](https://datatracker.ietf.org/doc/html/rfc9260), represented as 8 hex characters. +`SHA-256` | lowercase hex | As defined in [RFC 6234](https://datatracker.ietf.org/doc/html/rfc6234), represented as 64 hex characters. + +The `` encodings are: +- **lowercase hex**: the digest bytes rendered as lowercase hexadecimal, two characters per byte and no separators (for example, `MD5:d41d8cd98f00b204e9800998ecf8427e`). +- **opaque**: the token supplied verbatim by the object store, used only for equality comparison and not otherwise interpreted. `checksum` applies to the resolved bytes, except for `ETAG`, which is the object-store eTag for the whole file referenced by `path`. @@ -130,7 +134,7 @@ When File type is supported (`writerFeatures` field of a table's `protocol` acti - must write a column of type `file` to Parquet as a group annotated with the Parquet `FILE` logical type, whose fields are drawn from the set `path`, `offset`, `size`, `content_type`, `checksum`, and `inline`, with the Parquet primitive types described in [File data in Parquet](#file-data-in-parquet). The group need only define the fields it uses; any field it does define must be optional. - must not rename the fields within a `FILE`-annotated group; the field names above are normative. - must write every value so that it resolves to a referent according to [Byte Resolution](#byte-resolution). In particular, `size` must be set whenever `offset` is set, and a self-reference (`path` not set) must set `offset` (and therefore `size`). A value that does not resolve to any referent is invalid and must be represented as a column null instead. -- must, when writing a `checksum`, use the `:base64()` form with one of the recognized algorithms in [Checksum](#checksum). +- must, when writing a `checksum`, use the `:` form with one of the recognized algorithms and its digest encoding in [Checksum](#checksum). - must store additional metadata about a file (for example, a modification timestamp) adjacent to the `file` column, not inside the `FILE`-annotated group. ## Reader Requirements for File Data Type From 1b0746b1d3d3670f26c5218486b07768a300d4e6 Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Thu, 23 Jul 2026 13:22:30 +0000 Subject: [PATCH 07/10] [Protocol RFC] Sync File type with Parquet FILE spec (PR #585 @ 92228a1) Align with the latest apache/parquet-format#585 (commit 92228a1): - Update checksum RFC citations: MD5 -> RFC 1321, CRC32 -> RFC 2083, CRC32C -> RFC 3385 (SHA-256 unchanged at RFC 6234). - Readers may return a null file value for a row whose reference is invalid. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 5e20fd2969f..1acfad2f1f5 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -97,9 +97,9 @@ The `checksum` field is a self-describing token of the form `:` encodings are: @@ -143,6 +143,7 @@ When File type is supported (`readerFeatures` field of a table's `protocol` acti - must recognize and tolerate a `file` data type in a Delta schema. - must use the correct physical schema (a Parquet `FILE`-annotated group with the optional fields described in [File data in Parquet](#file-data-in-parquet)) when reading a `file` data type from a file. - must resolve each value to bytes according to [Byte Resolution](#byte-resolution), including the self-reference case (locating the bytes within the same data file when `path` is absent). +- may return a `null` `file` value for a row whose reference is invalid (does not resolve to any referent per [Byte Resolution](#byte-resolution)). - must make the column available to the engine: - [Recommended] Expose and interpret the group as a single `file` value, resolving inline, self-reference, and external bytes on access. - [Alternate] Expose the raw physical group (the set of present fields), for example if the engine does not natively support the `file` type. From 6f4afe88f8ad5948d5add4a83ac9bf5e6fb80712 Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Thu, 23 Jul 2026 16:14:24 +0000 Subject: [PATCH 08/10] [Protocol RFC] Rename File type path field to uri (Parquet PR #585 @ 4223744) Align with apache/parquet-format#585 (commit 4223744): the FILE locator field was renamed from `path` to `uri`. Rename the field throughout the field table, resolution table, Parquet examples, writer/reader requirements, statistics, and prose. Generic uses of "path" (field path, absolute path) are unchanged, as is the DBR-specific __FILE_TYPE_MODE storage-mode key. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 46 +++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 1acfad2f1f5..1b7985cdd54 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -16,8 +16,8 @@ The `file` data type is the Delta mapping of the Parquet `FILE` logical type pro This feature enables support for the `file` data type, which stores a reference to a range of bytes. A `file` value resolves to bytes that are located in one of three ways: - **inline** — the bytes are stored directly in the value, -- **self-reference** — the bytes are stored within the same data file that holds this `file` value, addressed by a byte range (`offset` / `size`) with no `path`. The bytes are written between column chunks and are not otherwise referenced by the Parquet footer, so a reader locates them within the current file rather than opening an external one. See [Byte Resolution](#byte-resolution). -- **external** — the bytes are stored in a separate file at a given `path`. +- **self-reference** — the bytes are stored within the same data file that holds this `file` value, addressed by a byte range (`offset` / `size`) with no `uri`. The bytes are written between column chunks and are not otherwise referenced by the Parquet footer, so a reader locates them within the current file rather than opening an external one. See [Byte Resolution](#byte-resolution). +- **external** — the bytes are stored in a separate file at a given `uri`. The schema serialization method is described in [Schema Serialization Format](#schema-serialization-format), and the physical encoding is described in [File data in Parquet](#file-data-in-parquet). @@ -53,26 +53,26 @@ To support this feature: ## File data in Parquet The `file` data type is represented in Parquet as a group annotated with the Parquet `FILE` logical type, as specified in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). -The group may contain the following fields, identified by name (matched case-sensitively, not by field order). Field IDs, if they exist, may also be used for projection. Every field is optional both in the schema and in the data: a writer may omit any field from the group definition, and any field that is present has repetition type `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only column may define just `inline`, and a whole-file external reference may define just `path`). +The group may contain the following fields, identified by name (matched case-sensitively, not by field order). Field IDs, if they exist, may also be used for projection. Every field is optional both in the schema and in the data: a writer may omit any field from the group definition, and any field that is present has repetition type `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only column may define just `inline`, and a whole-file external reference may define just `uri`). A field is *set* when it is present in the group and its value is non-null (and, for string fields, non-empty). A field is *not set* when it is absent from the group, or is present but null or empty. (Implementations are not expected to treat empty strings as null.) Struct field name | Parquet primitive type | Description -|-|- -path | binary (`STRING`) | A URI-reference as defined by [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986), which may be absolute or relative (for example, `s3://bucket/file.jpg`). No additional encoding (such as URI encoding) is applied on top of the user-provided value. If `path` is not set, the value refers to the current file (a self-reference). -offset | int64 | The start of the byte range within the referenced data. Must not be negative. If not set, readers must treat it as 0; if set and non-zero, readers must seek to this offset. `offset` must be set for a self-reference (`path` not set), and is optional for an external reference. -size | int64 | The byte length of the referenced data. Must be zero or a positive integer if set; 0 indicates empty referenced data. `size` must be set whenever `offset` is set. It may be omitted only for a whole-file external reference (`path` set, `offset` not set), in which case the range runs to the end of the referenced file. +uri | binary (`STRING`) | A URI-reference as defined by [RFC 3986](https://datatracker.ietf.org/doc/html/rfc3986), which may be absolute or relative (for example, `s3://bucket/file.jpg`). No additional encoding (such as URI encoding) is applied on top of the user-provided value. If `uri` is not set, the value refers to the current file (a self-reference). +offset | int64 | The start of the byte range within the referenced data. Must not be negative. If not set, readers must treat it as 0; if set and non-zero, readers must seek to this offset. `offset` must be set for a self-reference (`uri` not set), and is optional for an external reference. +size | int64 | The byte length of the referenced data. Must be zero or a positive integer if set; 0 indicates empty referenced data. `size` must be set whenever `offset` is set. It may be omitted only for a whole-file external reference (`uri` set, `offset` not set), in which case the range runs to the end of the referenced file. content_type | binary (`STRING`) | The media type (MIME type), as defined by [RFC 2046](https://datatracker.ietf.org/doc/html/rfc2046), of the resolved bytes, for example `image/png`. When not set, the type may be assumed to be `application/octet-stream`. checksum | binary (`STRING`) | A self-describing integrity token for the resolved bytes, of the form `:` (see [Checksum](#checksum)). -inline | binary | The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`path`, `offset`, `size`) that are set are provenance only. +inline | binary | The referenced bytes stored inline in the value. If `inline` is set, it supplies the bytes and any locator fields (`uri`, `offset`, `size`) that are set are provenance only. -A value resolves to bytes determined by `inline` / `path` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. The resolution rules are given in [Byte Resolution](#byte-resolution). +A value resolves to bytes determined by `inline` / `uri` / `offset` / `size`; `content_type` and `checksum` are metadata describing whatever is resolved. The resolution rules are given in [Byte Resolution](#byte-resolution). Because every field is optional, a group need only define the fields it uses. An example group that defines all fields: ``` optional group profile_image (FILE) { - optional binary path (STRING); + optional binary uri (STRING); optional int64 offset; optional int64 size; optional binary content_type (STRING); @@ -106,34 +106,34 @@ The `` encodings are: - **lowercase hex**: the digest bytes rendered as lowercase hexadecimal, two characters per byte and no separators (for example, `MD5:d41d8cd98f00b204e9800998ecf8427e`). - **opaque**: the token supplied verbatim by the object store, used only for equality comparison and not otherwise interpreted. -`checksum` applies to the resolved bytes, except for `ETAG`, which is the object-store eTag for the whole file referenced by `path`. +`checksum` applies to the resolved bytes, except for `ETAG`, which is the object-store eTag for the whole file referenced by `uri`. ### Byte Resolution -A value resolves to bytes based on which of `inline`, `path`, `offset`, and `size` are set. `size` must be set whenever `offset` is set, so every offset-based read carries an explicit `size`; `size` may be omitted only for a whole-file external reference. +A value resolves to bytes based on which of `inline`, `uri`, `offset`, and `size` are set. `size` must be set whenever `offset` is set, so every offset-based read carries an explicit `size`; `size` may be omitted only for a whole-file external reference. -`inline` | `path` | `offset` | `size` | Resolves to +`inline` | `uri` | `offset` | `size` | Resolves to :-:|:-:|:-:|:-:|- set | – | – | – | The `inline` bytes. -– | set | – | – | The whole external file at `path`. -– | set | – | set | External `path`, `[0, size)`. -– | set | set | set | External `path`, `[offset, offset + size)`. +– | set | – | – | The whole external file at `uri`. +– | set | – | set | External `uri`, `[0, size)`. +– | set | set | set | External `uri`, `[offset, offset + size)`. – | – | set | set | This file, `[offset, offset + size)` (self-reference). – | set | set | – | Invalid (`offset` set without `size`). – | – | set | – | Invalid (`offset` set without `size`). -– | – | – | set | Invalid (`size` set without `path` or `offset`). +– | – | – | set | Invalid (`size` set without `uri` or `offset`). – | – | – | – | Nothing — invalid (use column nullability for a null value). -A self-reference points within the same Parquet data file using `offset` and `size` (both required); the bytes are written between column chunks and are not otherwise referenced by the footer. A self-reference is the *absence* of `path`, never an absolute path back to the current file, so a data file containing self-references is renamed or relocated as a single unit. +A self-reference points within the same Parquet data file using `offset` and `size` (both required); the bytes are written between column chunks and are not otherwise referenced by the footer. A self-reference is the *absence* of `uri`, never an absolute path back to the current file, so a data file containing self-references is renamed or relocated as a single unit. -The referenced bytes (both `inline` and self-reference regions) are compressed with the same `CompressionCodec` as the one configured for the `inline` column; `size` is therefore the length of the compressed region on disk. External referents (`path` set) are opaque to Parquet and stored as-is. +The referenced bytes (both `inline` and self-reference regions) are compressed with the same `CompressionCodec` as the one configured for the `inline` column; `size` is therefore the length of the compressed region on disk. External referents (`uri` set) are opaque to Parquet and stored as-is. ## Writer Requirements for File Data Type When File type is supported (`writerFeatures` field of a table's `protocol` action contains `fileType`), writers: -- must write a column of type `file` to Parquet as a group annotated with the Parquet `FILE` logical type, whose fields are drawn from the set `path`, `offset`, `size`, `content_type`, `checksum`, and `inline`, with the Parquet primitive types described in [File data in Parquet](#file-data-in-parquet). The group need only define the fields it uses; any field it does define must be optional. +- must write a column of type `file` to Parquet as a group annotated with the Parquet `FILE` logical type, whose fields are drawn from the set `uri`, `offset`, `size`, `content_type`, `checksum`, and `inline`, with the Parquet primitive types described in [File data in Parquet](#file-data-in-parquet). The group need only define the fields it uses; any field it does define must be optional. - must not rename the fields within a `FILE`-annotated group; the field names above are normative. -- must write every value so that it resolves to a referent according to [Byte Resolution](#byte-resolution). In particular, `size` must be set whenever `offset` is set, and a self-reference (`path` not set) must set `offset` (and therefore `size`). A value that does not resolve to any referent is invalid and must be represented as a column null instead. +- must write every value so that it resolves to a referent according to [Byte Resolution](#byte-resolution). In particular, `size` must be set whenever `offset` is set, and a self-reference (`uri` not set) must set `offset` (and therefore `size`). A value that does not resolve to any referent is invalid and must be represented as a column null instead. - must, when writing a `checksum`, use the `:` form with one of the recognized algorithms and its digest encoding in [Checksum](#checksum). - must store additional metadata about a file (for example, a modification timestamp) adjacent to the `file` column, not inside the `FILE`-annotated group. @@ -142,7 +142,7 @@ When File type is supported (`writerFeatures` field of a table's `protocol` acti When File type is supported (`readerFeatures` field of a table's `protocol` action contains `fileType`), readers: - must recognize and tolerate a `file` data type in a Delta schema. - must use the correct physical schema (a Parquet `FILE`-annotated group with the optional fields described in [File data in Parquet](#file-data-in-parquet)) when reading a `file` data type from a file. -- must resolve each value to bytes according to [Byte Resolution](#byte-resolution), including the self-reference case (locating the bytes within the same data file when `path` is absent). +- must resolve each value to bytes according to [Byte Resolution](#byte-resolution), including the self-reference case (locating the bytes within the same data file when `uri` is absent). - may return a `null` `file` value for a row whose reference is invalid (does not resolve to any referent per [Byte Resolution](#byte-resolution)). - must make the column available to the engine: - [Recommended] Expose and interpret the group as a single `file` value, resolving inline, self-reference, and external bytes on access. @@ -165,10 +165,10 @@ Change Data Feed | **Supported:** A table using the `file` data type is allowed A `file` value is physically a group of leaf fields (see [File data in Parquet](#file-data-in-parquet)), and Delta's [Per-file Statistics](#per-file-statistics) are already encoded mirroring the schema of the data, descending into nested fields. Statistics for a `file` column follow that same per-leaf model, with one exception for the `inline` field: - The `nullCount` statistic is collected for the `file` column itself (whether the whole `file` value is null), following the standard nested-field statistics encoding. -- `minValues` and `maxValues` are collected per leaf field, for the comparable leaf fields only: `path` (STRING), `offset` (INT64), `size` (INT64), `content_type` (STRING), and `checksum` (STRING). These follow the standard rules for their respective types (for example, STRING leaves such as `path` are truncated to a fixed prefix length, as with any string column). +- `minValues` and `maxValues` are collected per leaf field, for the comparable leaf fields only: `uri` (STRING), `offset` (INT64), `size` (INT64), `content_type` (STRING), and `checksum` (STRING). These follow the standard rules for their respective types (for example, STRING leaves such as `uri` are truncated to a fixed prefix length, as with any string column). - `minValues` and `maxValues` are **not** collected for the `inline` field, because it is binary content for which min/max provides no data-skipping value and may be large. -Collecting `minValues` / `maxValues` on `path` in particular enables data skipping on file-inventory and manifest tables that filter by path (for example, an object-store prefix). +Collecting `minValues` / `maxValues` on `uri` in particular enables data skipping on file-inventory and manifest tables that filter by URI (for example, an object-store prefix). The set of columns for which statistics are collected is otherwise governed by the table's existing statistics configuration (for example, the number of indexed columns). From ef13d87b1bd04d54758b963520e2488925b3332c Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Fri, 24 Jul 2026 12:46:41 +0000 Subject: [PATCH 09/10] [Protocol RFC] Sync File type with Parquet FILE spec (PR #585 @ f241be8) Align with apache/parquet-format#585 (commit f241be8): add the modular-encryption restriction -- data files containing self-references must not use Parquet modular encryption, since self-referenced byte ranges are not encryption modules and cannot be encrypted/authenticated independently. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 1b7985cdd54..65dbaa6d652 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -128,6 +128,8 @@ A self-reference points within the same Parquet data file using `offset` and `si The referenced bytes (both `inline` and self-reference regions) are compressed with the same `CompressionCodec` as the one configured for the `inline` column; `size` is therefore the length of the compressed region on disk. External referents (`uri` set) are opaque to Parquet and stored as-is. +Parquet data files containing self-references must not use Parquet modular encryption: self-referenced byte ranges are not Parquet encryption modules and therefore cannot be encrypted or authenticated independently. Encryption of external files referenced by `uri` is outside the scope of the Parquet format (and thus of this feature). + ## Writer Requirements for File Data Type When File type is supported (`writerFeatures` field of a table's `protocol` action contains `fileType`), writers: From 73e82040dd63877062e9fa5bd0b257eff9f56ddd Mon Sep 17 00:00:00 2001 From: dejankrak-db <184246885+dejankrak-db@users.noreply.github.com> Date: Sun, 26 Jul 2026 17:02:44 +0000 Subject: [PATCH 10/10] [Protocol RFC] Reference merged Parquet FILE spec (PR #585 merged) apache/parquet-format#585 is now merged. The FILE spec content is unchanged from the last sync (commit f241be8), so no semantic updates are needed. Update references to point at the stable LogicalTypes.md#file spec and drop the "proposed" framing now that the type is part of the Parquet format. Co-authored-by: Isaac --- protocol_rfcs/file-type.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/protocol_rfcs/file-type.md b/protocol_rfcs/file-type.md index 65dbaa6d652..e6afb049d99 100644 --- a/protocol_rfcs/file-type.md +++ b/protocol_rfcs/file-type.md @@ -5,7 +5,7 @@ This protocol change adds support for the `file` data type. The `file` data type stores a reference to a range of bytes that may be located inline in the value, elsewhere within the same data file, or in an external file. It is intended for use cases such as file inventories, manifests, and unstructured-data references (for example, images or audio stored in object storage), which are increasingly common with AI/ML workloads. -The `file` data type is the Delta mapping of the Parquet `FILE` logical type proposed in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). This RFC is aligned with that proposal: the physical Parquet representation, the field set, and the byte-resolution rules defined here match the Parquet `FILE` type so that a Delta `file` column round-trips through Parquet without loss. +The `file` data type is the Delta mapping of the Parquet [`FILE` logical type](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file) (introduced in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585)). This RFC is aligned with that specification: the physical Parquet representation, the field set, and the byte-resolution rules defined here match the Parquet `FILE` type so that a Delta `file` column round-trips through Parquet without loss. -------- @@ -52,7 +52,7 @@ To support this feature: ## File data in Parquet -The `file` data type is represented in Parquet as a group annotated with the Parquet `FILE` logical type, as specified in [apache/parquet-format#585](https://github.com/apache/parquet-format/pull/585). +The `file` data type is represented in Parquet as a group annotated with the Parquet `FILE` logical type, as specified in [Parquet LogicalTypes.md](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file). The group may contain the following fields, identified by name (matched case-sensitively, not by field order). Field IDs, if they exist, may also be used for projection. Every field is optional both in the schema and in the data: a writer may omit any field from the group definition, and any field that is present has repetition type `OPTIONAL`. A group need only define the fields it uses (for example, an inline-only column may define just `inline`, and a whole-file external reference may define just `uri`). A field is *set* when it is present in the group and its value is non-null (and, for string fields, non-empty). A field is *not set* when it is absent from the group, or is present but null or empty. (Implementations are not expected to treat empty strings as null.)