Skip to content

Introduce FileType reference implementation - #10109

Open
brkyvz wants to merge 4 commits into
apache:mainfrom
brkyvz:fileType
Open

Introduce FileType reference implementation#10109
brkyvz wants to merge 4 commits into
apache:mainfrom
brkyvz:fileType

Conversation

@brkyvz

@brkyvz brkyvz commented Jun 10, 2026

Copy link
Copy Markdown

Which issue does this PR close?

TODO

  • Closes #NNN.

Rationale for this change

Introduces the reference implementation for the new LogicalTypeAnnotation File as discussed in this design document

What changes are included in this PR?

Introduces a new LogicalTypeAnnotation called FILE. It enforces that a group with this type annotation have the following fields by name:

uri 		 		STRING	
offset				INT64      
size	 	 		INT64	
content_type		 STRING
checksum		 	STRING
inline 		 		BINARY

Are these changes tested?

Yes, unit tested

Are there any user-facing changes?

Introduces a new LogicalTypeAnnotation called FILE

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Jun 10, 2026
@etseidl etseidl added the api-change Changes to the arrow API label Jun 10, 2026

@etseidl etseidl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just one test nit.

I marked this as a breaking change since it adds a variant to a public enum.

Comment thread parquet/src/schema/types.rs Outdated
assert!(result
.unwrap_err()
.to_string()
.contains("unrecognized field"));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: in general I prefer full string compares to contains. In this case it should at least test for the specific unrecognized field name 'unknown_field'.

@etseidl

etseidl commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Looking very nice. Could you also add a roundtrip test to/from parquet? 🙏

Also, do you have any plans to add a FILE containing file to parquet-testing?

@etseidl etseidl added the next-major-release the PR has API changes and it waiting on the next major version label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api-change Changes to the arrow API next-major-release the PR has API changes and it waiting on the next major version parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants