1.3 Release planning

[yihua]

Starting this discussion to plan out the 1.3 release. I propose this release to focus on advancing features in unified data foundation for analytics and AI, and engine integration and improvements to new unstructured support. See separate comments on tracks and features.

Proposed timeline: code freeze by TBD

[yihua]

Storage Format

• Parquet Format Tuning for VECTOR, BLOB, and VARIANT (?) with micro benchmarks
• Column Family
• Vortex File Format support Vortex File Format support in Hudi #18623
• Improvements and bug fixes on unstructured types
• Shredded support for VARIANT

[yihua]

@rahil-c @voonhous to confirm

[voonhous]

Added Shredded support for variant.

[vinothchandar]

Added items from the 1.4 discussion

[yihua]

Spark

• Spark 4.2 support feat(spark): add Spark 4.2 support #18621
• Use engine-native record representation on Spark writers

[suryaprasanna]

@yihua Can we please remove the "Spark 3.3 deprecation" deliverable from 1.3 release?
Attaching the corresponding issue for reference here, #18917

[yihua]

"Spark 3.3 deprecation" is removed and deferred to future releases.

[yihua]

Flink

• Flink 1.17 deprecation
• Flink 2.2 support: Support Flink 2.2 #18912
• Variant data type support: Add Flink support for HoodieSchema Variant data type #18513
• Blob data type support: Add Flink support for Blob data type #18507
• Vector data type support: Add Flink support for HoodieSchema Vector type #18506
• Lance format integration: Lance format support for Flink #18800
• LSM layout(?): New LSM file layout #14310

[yihua]

@danny0405 @cshuo to add

[cshuo]

Support Flink 2.2 for more functionalities, like VECTOR_SEARCH.

[yihua]

Presto and Trino

• Hudi 1.2 upgrade in Presto connector
• Hudi 1.2 upgrade in Trino connector
• Trino connector refactoring
  • [FEATURE] Migrate Trino Hudi connector internals to a Hudi-published hudi-trino artifact (shim/bundle refactor) #18780
  • feat(trino): [RFC-105] Trino Hudi Connector — Shim/Bundle Refactor #18782
  • feat(trino): Migrate the Trino-Hudi connector into the Hudi repo (RFC-105) #18837
• (Stretch) Unstructured type support

[yihua]

@voonhous to add

[yihua]

Storage Engine

• Vector Search Algorithm improvements
• Vector Index RFC-104: Add Vector Index Implementation within Hudi #18676
• Managed Blobs - blob cleaning, managed in-table blobs
• Branch and tag
• Metadata table improvements - indexer abstraction, performance
• (stretch) Schema system improvements
• (stretch) New indexes - B-tree, Bitmap, etc.
• (stretch) Concurrent updates during clustering
• (stretch) Table service improvement

[danny0405]

1. data evolution: this is very important for AI feature engineering to add feature column with flexibility(required by JD and Kuaishou);
2. branch & tag: also required by AI feature engineering to try new features(which is in experiment and once confirmed, these columns would be put into production table), pull out a new branch for specific feature is a natural solution here (required by JD and Kuaishou);

Please improve the priorities of these two, they are required for the AI stories.

[danny0405]

I want to fire a dicsussion to remove the metadata fields, keeping the _hoodie_instant_time for incremental query purposes, all the other existing metadata fields can be inferred on the fly on reader side. It does not break compatibility, still deem the existing metadata columns as virtual in Hudi(there is no physical persist except for the instant time).

[codope]

+1 for branch & tag. it's a very useful feature. Even I want to explore the idea for shadow testing and validation framework.

[codope]

@nsivabalan @yihua anyone actively working on #14611 ?
I recall a strawman proposal from @vinothchandar couple years back. We whiteboarded but it never took off. I think with completion time based file slicing, it might be a good time to revisit it. Also, in the SIGMOD workshop I came across a beautiful paper on "commutative compaction" (which is already true in Hudi) and those principles can be applied for clustering too.

[danny0405]

@codope the paper looks awesome!

[nsivabalan]

Platform Components:

• Hudi Streamer support for multimodal data: vector embeddings, blob management, etc.
• (Stretch) Hudi CLI MCP Server: Adding hudi-cli-mcp to assist AI assistants to help and manage hudi tables #18554
• (Stretch)Hudi Timeline UI: feat(ui): [HUDI-9315] Add RFC-94: Hudi Timeline UI #13152 feat(timeline-ui): [HUDI-9315] Add Hudi Timeline UI #13147

[voonhous]

Added hudi-ui