[Protocol RFC] File data type by dejankrak-db · Pull Request #7148 · delta-io/delta

dejankrak-db · 2026-07-06T23:44:32Z

Which Delta project/connector is this regarding?

Description

This adds a protocol RFC for a new file data type. A file value stores a reference to a range of bytes that may be located inline in the value, in a self-reference elsewhere within the same data file, or in an external file at a given URI. It targets file-inventory, manifest, and unstructured-data use cases (e.g. images/audio in object storage) that are increasingly common with AI/ML workloads.

The type is aligned with the Parquet FILE logical type (introduced in apache/parquet-format#585, now merged): the field set (uri, offset, size, content_type, checksum, inline), the physical encoding, and the byte-resolution rules match the Parquet spec so a Delta file column round-trips through Parquet without loss.

Design doc: [DESIGN] Introducing a new "File" logical type to Parquet

The RFC introduces:

a new fileType table feature (Reader Version 3 / Writer Version 7, readers and writers);
a new file primitive type name in the schema serialization format;
the Parquet physical encoding (a group annotated with the FILE logical type, all six optional fields) and byte-resolution rules, including inline, self-reference, and external references;
per-leaf statistics — nullCount on the column and minValues/maxValues on the comparable leaf fields (uri, offset, size, content_type, checksum), excluding inline — enabling data skipping on file-inventory tables filtered by URI;
an optional per-column storage mode (__FILE_TYPE_MODE: MANAGED / EXTERNAL / UNKNOWN) recorded in schema metadata, mirroring the __COLLATIONS representation; only the representation is standardized, with lifecycle/GC semantics left as a non-goal;
time travel and Change Data Feed semantics — Delta time-travels/CDFs the reference, but makes no guarantee about the referenced bytes, which live outside the transaction log;
compatibility notes with partitioning, clustering, generated columns, CHECK constraints, default values, and Change Data Feed.

This is a documentation-only change adding protocol_rfcs/file-type.md and listing it in protocol_rfcs/README.md.

see #7147

Add a protocol RFC for a new `file` data type that stores a reference to a range of bytes located inline, elsewhere in the same data file, or in an external file. Aligned with the Parquet FILE logical type proposed in apache/parquet-format#585. Introduces the `fileType` table feature (Reader v3 / Writer v7), the `file` primitive type name, Parquet physical encoding, per-leaf statistics, and compatibility notes with other Delta features. See delta-io#7147 Co-authored-by: Isaac

ilicmarkodb · 2026-07-07T14:17:04Z

+Partition Columns | **Supported:** A `file` column is allowed to be a non-partitioned column of a partitioned table. <br/> **Unsupported:** A `file` value is a group and cannot be serialized to a partition-value string, so a `file` column cannot be a partition column.
+Clustered Tables | **Supported:** A `file` column is allowed to be a non-clustering column of a clustered table. <br/> **Unsupported:** A `file` value is a group and is not a comparable data type as a whole, so a `file` column cannot be a clustering column.
+Delta Column Statistics | **Supported:** A `file` column supports the `nullCount` statistic, and `minValues` / `maxValues` on its comparable leaf fields. See [Statistics for File Columns](#statistics-for-file-columns). <br/> **Unsupported:** The `file` column as a whole is not a comparable data type, and the `inline` field does not support `minValues` / `maxValues`.
+Generated Columns | **Supported:** A `file` column is allowed to be used as a source in a generated column expression, as long as the `file` type is not the result type of the generated column expression. <br/> **Unsupported:** The `file` data type is not allowed to be the result type of a generated column expression.


why wouldn't we have FILE generated columns?

Fair question — that restriction was carried over from the Variant RFC. I've reframed this as an open question for discussion on the issue.

…-columns question - Expand the self-reference bullet in the intro with how the bytes are located within the same data file, linking to Byte Resolution. - Reframe the generated-column result-type clause as an open question for discussion rather than a hard restriction inherited from the variant RFC. Co-authored-by: Isaac

…on-goals - Add a "Time Travel and Change Data Feed" section clarifying that Delta time- travels and CDFs the reference, not the referenced bytes, which live outside the transaction log and may be overwritten or deleted independently. - Add a "Non-Goals" section scoping out lifecycle/GC of referenced bytes (including any Delta-managed notion and its VACUUM interaction) and access brokering, keeping the type reference-only and aligned with Parquet/Iceberg. Co-authored-by: Isaac

Standardize the representation of a file column's storage mode (MANAGED/EXTERNAL/UNKNOWN) in the __FILE_TYPE_MODE schema-metadata key, a path-keyed map on the nearest ancestor StructField, mirroring the __COLLATIONS representation and DBR's FileType mode (databricks-eng/runtime#213587). This lets the qualifier round-trip through the Delta log without placing it inside the file value (which must round-trip through the fixed Parquet FILE field set). Only the representation is standardized; MANAGED vs EXTERNAL carries no Delta behavior yet, and lifecycle/GC remains a non-goal. Co-authored-by: Isaac

…a-io#585) Align with the latest apache/parquet-format#585: - size must be set whenever offset is set; a self-reference (no path) must set offset, and therefore size. Drop the now-invalid [offset, EOF) and [0, size) self-reference modes and the external [offset, EOF) mode from the resolution table; add explicit invalid rows. - Define "set" (present, non-null, non-empty for strings) and allow sparse group definitions (a group need only define the fields it uses); add an inline-only example group. - Fields matched case-sensitively by name; field IDs "if they exist". - Readers should ignore unknown checksum algorithms. Follows the consistent prose intent of PR delta-io#585; note its resolution table still lists an [offset, EOF) row that contradicts its own validation section (offset requires size) -- to be raised on the Parquet PR. Co-authored-by: Isaac

…@ 90147dc) Align with the latest apache/parquet-format#585 (commit 90147dc): - path is now a URI-reference per RFC 3986 (absolute or relative), not an opaque location string. Resolves the path/uri debate: name stays `path`, semantics become URI. - checksum form changes from <algorithm>:base64(<digest>) to <algorithm>:<digest> with per-algorithm encoding (lowercase hex for MD5/CRC32/CRC32C/SHA-256, opaque for ETAG); add RFC references and an Encoding column. - content_type: MIME per RFC 2046, defaults to application/octet-stream when unset. - offset must not be negative. - Note that implementations are not expected to treat empty strings as null. Co-authored-by: Isaac

…@ 92228a1) Align with the latest apache/parquet-format#585 (commit 92228a1): - Update checksum RFC citations: MD5 -> RFC 1321, CRC32 -> RFC 2083, CRC32C -> RFC 3385 (SHA-256 unchanged at RFC 6234). - Readers may return a null file value for a row whose reference is invalid. Co-authored-by: Isaac

…o#585 @ 4223744) Align with apache/parquet-format#585 (commit 4223744): the FILE locator field was renamed from `path` to `uri`. Rename the field throughout the field table, resolution table, Parquet examples, writer/reader requirements, statistics, and prose. Generic uses of "path" (field path, absolute path) are unchanged, as is the DBR-specific __FILE_TYPE_MODE storage-mode key. Co-authored-by: Isaac

…@ f241be8) Align with apache/parquet-format#585 (commit f241be8): add the modular-encryption restriction -- data files containing self-references must not use Parquet modular encryption, since self-referenced byte ranges are not encryption modules and cannot be encrypted/authenticated independently. Co-authored-by: Isaac

…rged) apache/parquet-format#585 is now merged. The FILE spec content is unchanged from the last sync (commit f241be8), so no semantic updates are needed. Update references to point at the stable LogicalTypes.md#file spec and drop the "proposed" framing now that the type is part of the Parquet format. Co-authored-by: Isaac

dejankrak-db marked this pull request as ready for review July 7, 2026 09:39

ilicmarkodb reviewed Jul 7, 2026

View reviewed changes

dejankrak-db added 9 commits July 7, 2026 23:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Protocol RFC] File data type - #7148

[Protocol RFC] File data type#7148
dejankrak-db wants to merge 10 commits into
delta-io:masterfrom
dejankrak-db:dejankrak-db/file-type-rfc

dejankrak-db commented Jul 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

ilicmarkodb Jul 7, 2026

Uh oh!

dejankrak-db Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

dejankrak-db commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

Uh oh!

Uh oh!

ilicmarkodb Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

dejankrak-db Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dejankrak-db commented Jul 6, 2026 •

edited

Loading