HoodieSchema collapses TINYINT/SMALLINT into INT and loses engine type width needed by writer paths

[cshuo]

Task Description

Describe the problem

HoodieSchema currently collapses engine-level small integer types into a single INT type, which loses integer width information needed to faithfully reconstruct engine-native schemas.

Examples:

• Spark: ByteType | ShortType -> HoodieSchemaType.INT
• Flink: TINYINT | SMALLINT -> HoodieSchemaType.INT

This becomes a problem because writer paths are built from HoodieSchema, and later reconstruct engine-native schemas from it.

On Spark, the path is roughly:

1. HoodieSparkSchemaConverters.toHoodieType(...) maps ByteType/ShortType/IntegerType to HoodieSchemaType.INT
2. HoodieSparkFileWriterFactory reconstructs StructType from HoodieSchema
3. HoodieSparkSchemaConverters.toSqlType(...) maps HoodieSchemaType.INT back to IntegerType
4. HoodieRowParquetWriteSupport makes type-dependent row accessor decisions from that reconstructed StructType

At that point, the original Spark type width is already lost. For example, an original ShortType field is reconstructed as IntegerType, and writer code may go through row.getInt(...) instead of row.getShort(...).

Flink has the same issue in principle:

• TINYINT/SMALLINT are also collapsed into HoodieSchemaType.INT
• writer code later reconstructs RowType from HoodieSchema
• the original integer width is no longer recoverable from HoodieSchema alone

So this is not just a schema round-trip fidelity issue. HoodieSchema currently does not preserve enough information for engine-native writer construction.

To Reproduce

1. Create a Spark StructType or Flink RowType containing TINYINT or SMALLINT
2. Convert it to HoodieSchema
3. Reconstruct engine-native schema from that HoodieSchema
4. Observe that the field has already been widened to INT / IntegerType

Expected behavior

HoodieSchema should preserve integer width information so that:

• Spark ByteType, ShortType
• Flink TINYINT, SMALLINT

do not all collapse into the same schema type.

Environment Description

Hudi version: current master / current branch

Possible solutions

1. Add new types:
   add native integer-width types such as TINYINT and SMALLINT to HoodieSchema, and preserve them across Spark/Flink converters.

2. Fix the writer builders:
   avoid reconstructing engine writer schema from HoodieSchema alone in paths that need engine-native getter semantics, and instead pass the original engine schema through the writer boundary.

The first direction seems more consistent with the long-term schema-system design. rfc-99 already describes primitive integer widths such as TINYINT and SMALLINT as first-class logical types.

Task Type

Code improvement/refactoring

Related Issues

Parent feature issue: (if applicable )
Related issues:
NOTE: Use Relationships button to add parent/blocking issues after issue is created.

[danny0405]

+1

[cshuo]

cc @voonhous @rahil-c do you have any thoughts or context?

[cshuo]

1. ANSI / ISO SQL Standard (2023)

Type	Standard	Typical meaning
TINYINT	No	Non-standard extension, usually an 8-bit signed integer
SMALLINT	Yes	Standard exact numeric type, usually implemented as a 16-bit signed integer

2. Big Data Engines and Apache Arrow

System	TINYINT	SMALLINT	Type mapping
Apache Hive	Yes	Yes	8-bit / 16-bit signed integer
Apache Spark SQL	Yes	Yes	ByteType / ShortType
Apache Flink SQL	Yes	Yes	Java Byte / Short
Trino / Presto	Yes	Yes	8-bit / 16-bit signed integer
Apache Impala	Yes	Yes	1-byte / 2-byte signed integer
Apache Doris	Yes	Yes	8-bit / 16-bit signed integer
StarRocks	Yes	Yes	1-byte / 2-byte signed integer
Apache Arrow	Int8	Int16	Uses explicit bit-width types instead of SQL type names

[voonhous]

Had a chat with @cshuo this afternoon SGT and the reason we did not add TINYINT/SMALLINT into HoodieSchema was because we wanted a 1:1 mapping between Avro and HoodieSchema during the refactor.

[danny0405]

@cshuo what changes we need to make if TINYINT and SMALLINT are introduced, for e.g, do we need fixes to the col_stats?

[cshuo]

@cshuo what changes we need to make if TINYINT and SMALLINT are introduced, for e.g, do we need fixes to the col_stats?

Yes, col_stats will need a small follow-up, but a new stats type is not necessarily required. TINYINT / SMALLINT can continue to use the existing integer stats path; we just need the new schema types to be recognized there.

More broadly, supporting TINYINT / SMALLINT involves at least these areas:

1. Shared schema model
   HoodieSchema / HoodieSchemaType needs to represent TINYINT and SMALLINT explicitly, while keeping compatible underlying storage.

2. Engine schema converters
   Spark, Flink and other engine converters need to preserve width when converting TINYINT / SMALLINT types.

3. Writer schema reconstruction
   Any writer path that rebuilds engine-native schema from HoodieSchema needs to round-trip these types correctly, otherwise they still get widened back to INT.

4. Metadata / stats / utility handling
   Places that switch on HoodieSchemaType for metadata handling, comparisons, defaults, partition parsing, or column stats need to recognize the new types and route them through the appropriate existing integer behavior.

5. Existing table compatibility
   The physical storage risk is relatively small, since this is still backed by the same underlying integer family with extra logical-width information on the Hudi schema side.
   The bigger risk is schema semantics for existing tables. Older tables already record these fields as INT in HoodieSchema. Once new code starts materializing TINYINT / SMALLINT explicitly, the system will see a schema difference where previously it saw the same type. That can affect schema validation, schema evolution checks, and concurrent schema conflict resolution.

6. Schema evolution semantics
   This likely needs explicit rules. The tricky part is that an existing INT in table schema is ambiguous: it may be a real INT, or it may be a legacy-collapsed TINYINT / SMALLINT. So we cannot blindly treat INT -> SMALLINT/TINYINT as a universally safe evolution rule.

7. Compatibility and tests
   We need round-trip and writer-path coverage for both Spark and Flink, plus targeted checks for metadata-related paths such as column stats, as well as validation around existing-table behavior and schema evolution.

So my current view is:

• col_stats is a small follow-up, the main work is not just schema enums, but making sure width survives the full path from engine schema to HoodieSchema to reconstructed engine schema to writer
• the main design sensitivity is around existing tables and schema evolution, because old INT does not tell us whether it was originally a true INT or a collapsed narrower engine type

[cshuo]

I dug into the Spark writer implementation a bit more, and find that the Spark write path is actually not broken for ByteType/ShortType.

Even though HoodieSchema collapses them into INT, the Spark writer widens byte/short → int as part of the normal record-rewrite step before the actual writing, so the stored row, data-file schema, and read path stay consistently IntegerType — no getInt()/getShort() mismatch.

hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieCreateRecordUtils.scala

Lines 189 to 197 in 1a8c8ad

	// NOTE: To make sure we properly transform records
	val finalStructTypeRowWriter = getCachedUnsafeRowWriter(sourceStructType, finalStructType)
	it.map { sourceRow =>
	val (key: HoodieKey, recordLocation: Option[HoodieRecordLocation]) =
	HoodieCreateRecordUtils.getHoodieKeyAndMayBeLocationFromSparkRecord(sparkKeyGenerator, sourceRow, sourceStructType,
	preppedSparkSqlWrites || preppedWriteOperation, preppedSparkSqlWrites || preppedWriteOperation || preppedSparkSqlMergeInto)
	val targetRow = finalStructTypeRowWriter(sourceRow)
	val hoodieSparkRecord = new HoodieSparkRecord(key, targetRow, dataFileStructType, false)

(Short/Byte value in sourceRow are rewritten into Int value in targetRow)

In other words, on the Spark side this is a type-fidelity characteristic (ShortType round-trips back as IntegerType), not a write-correctness bug.

Given that, We have a 3rd option that's narrower than the two above: Have the Flink writer do the same pre-write RowData rewrite & widen.

• Write: insert a RowData conversion (natural hook: RowDataToHoodieFunction, plus the bulk-insert path) that reads byte/short and widens them to int into a new RowData built against the target RowType reconstructed from the schema (already INT). After this, RowData / parquet / Avro schema are all consistently INT, so any code path that rebuilds a RowType from the schema and accesses the RowData no longer hits a getInt()-on-physical-short ClassCastException.
• Read: the read RowType is already derived from the Avro schema (INT), so the column naturally comes back as int, and we need rewrite the int value to origin SMALLINT/SHORTINT type before emitting to downstreams.

This keeps HoodieSchema unchanged and aligns Flink's behavior with Spark — TINYINT/SMALLINT become writable/readable, stored and returned as INT. Worth noting: the rewrite/widen only kicks in when the schema actually contains TINYINT/SMALLINT — for tables without these types the conversion is a no-op (or can be skipped entirely), so there's no extra overhead in the common case.

[danny0405]

Let's be conservative to adding new types, and try to flind a elegant solution to fix the read/write type consistency.