Protocol Change Request
Description of the protocol change
This proposes adding a new file data type to the Delta protocol. A file value stores a reference to a range of bytes that may be located inline in the value, in a self-reference elsewhere within the same data file, or in an external file at a given URI. It is intended for use cases such as file inventories, manifests, and unstructured-data references (for example, images or audio stored in object storage), which are increasingly common with AI/ML workloads.
The file data type is the Delta mapping of the Parquet FILE logical type (introduced in apache/parquet-format#585, now merged). This RFC is aligned with that specification: the physical Parquet representation, the field set (uri, offset, size, content_type, checksum, inline), and the byte-resolution rules match the Parquet FILE type so that a Delta file column round-trips through Parquet without loss.
Design doc: [DESIGN] Introducing a new "File" logical type to Parquet — https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit
Highlights of the proposed protocol change:
- New table feature
fileType (Reader Version 3 / Writer Version 7), for readers and writers.
- New primitive type name
file in the schema serialization format.
- Physical encoding as a Parquet group annotated with the
FILE logical type, with all six optional fields, covering inline, self-reference, and external references.
- Per-leaf statistics:
nullCount on the column, and minValues/maxValues on the comparable leaf fields (uri, offset, size, content_type, checksum), excluding inline. This enables data skipping on file-inventory tables filtered by URI.
- An optional per-column storage mode (
__FILE_TYPE_MODE: MANAGED / EXTERNAL / UNKNOWN) recorded in schema metadata, mirroring the __COLLATIONS representation; only the representation is standardized, with lifecycle/GC semantics left as a non-goal.
- Time travel and Change Data Feed semantics: Delta time-travels/CDFs the reference, but makes no guarantee about the referenced bytes, which live outside the transaction log.
- Compatibility notes for partitioning, clustering, generated columns, CHECK constraints, default values, and Change Data Feed.
An RFC document PR accompanies this issue (see the linked PR). Per the RFC process, this issue is the central place for discussion.
Willingness to contribute
The Delta Lake Community encourages protocol innovations. Would you or another member of your organization be willing to contribute this feature to the Delta Lake code base?
Protocol Change Request
Description of the protocol change
This proposes adding a new
filedata type to the Delta protocol. Afilevalue stores a reference to a range of bytes that may be located inline in the value, in a self-reference elsewhere within the same data file, or in an external file at a given URI. It is intended for use cases such as file inventories, manifests, and unstructured-data references (for example, images or audio stored in object storage), which are increasingly common with AI/ML workloads.The
filedata type is the Delta mapping of the ParquetFILElogical type (introduced in apache/parquet-format#585, now merged). This RFC is aligned with that specification: the physical Parquet representation, the field set (uri,offset,size,content_type,checksum,inline), and the byte-resolution rules match the ParquetFILEtype so that a Deltafilecolumn round-trips through Parquet without loss.Design doc: [DESIGN] Introducing a new "File" logical type to Parquet — https://docs.google.com/document/d/1AiwrstqkwkBoOZqgOkm9JGwSMcNeHyLR7EEj1CVqpZQ/edit
Highlights of the proposed protocol change:
fileType(Reader Version 3 / Writer Version 7), for readers and writers.filein the schema serialization format.FILElogical type, with all six optional fields, covering inline, self-reference, and external references.nullCounton the column, andminValues/maxValueson the comparable leaf fields (uri,offset,size,content_type,checksum), excludinginline. This enables data skipping on file-inventory tables filtered by URI.__FILE_TYPE_MODE:MANAGED/EXTERNAL/UNKNOWN) recorded in schema metadata, mirroring the__COLLATIONSrepresentation; only the representation is standardized, with lifecycle/GC semantics left as a non-goal.An RFC document PR accompanies this issue (see the linked PR). Per the RFC process, this issue is the central place for discussion.
Willingness to contribute
The Delta Lake Community encourages protocol innovations. Would you or another member of your organization be willing to contribute this feature to the Delta Lake code base?