Background
Spark 4 added lossless Parquet type widening in SPARK-40876. The current UINT64 compatibility fix in Bolt is intentionally limited to Spark's dedicated mapping:
Parquet UINT64 -> Spark DECIMAL(20, 0)
General integer-to-Decimal evolution should be implemented separately because it requires value rescaling and end-to-end handling beyond the schema compatibility predicate.
Spark 4 behavior to match
Decimal to Decimal
Spark 4 allows:
DECIMAL(p1, s1) -> DECIMAL(p2, s2)
when:
s2 >= s1
p2 - p1 >= s2 - s1
Equivalently, the target cannot reduce scale or the number of integer digits. Scale increases are materialized by rescaling the unscaled value.
Signed integer to Decimal
Spark 4 treats unannotated or signed Parquet integers as scale-zero values and allows:
Parquet INT32 -> DECIMAL(p, s) when p - s >= 10
Parquet INT64 -> DECIMAL(p, s) when p - s >= 20
The rule follows the Parquet physical type. Byte and Short Spark columns are also physically INT32, so they require the INT32 bound. Non-zero target scales require rescaling.
Unsigned integers are not part of this generic signed-integer widening rule. UINT64 keeps its dedicated DECIMAL(20, 0) representation path.
Bolt work needed
- Implement and test the Spark 4 compatibility rules without broadening Spark 3.5 behavior unintentionally.
- Rescale values when the target Decimal scale is larger.
- Handle short- and long-Decimal storage transitions correctly.
- Cover plain and dictionary encodings, nulls, and boundary values.
- Ensure filters, row-group statistics, metadata filters, and aggregation/value-hook pushdown are either converted correctly or safely disabled while preserving residual filtering.
- Verify error/fallback behavior for unsupported or narrowing conversions.
- Audit Bolt's existing Decimal-to-Decimal compatibility predicate and materialization path for complete Spark 4 parity.
References
Background
Spark 4 added lossless Parquet type widening in SPARK-40876. The current UINT64 compatibility fix in Bolt is intentionally limited to Spark's dedicated mapping:
General integer-to-Decimal evolution should be implemented separately because it requires value rescaling and end-to-end handling beyond the schema compatibility predicate.
Spark 4 behavior to match
Decimal to Decimal
Spark 4 allows:
when:
Equivalently, the target cannot reduce scale or the number of integer digits. Scale increases are materialized by rescaling the unscaled value.
Signed integer to Decimal
Spark 4 treats unannotated or signed Parquet integers as scale-zero values and allows:
The rule follows the Parquet physical type. Byte and Short Spark columns are also physically INT32, so they require the INT32 bound. Non-zero target scales require rescaling.
Unsigned integers are not part of this generic signed-integer widening rule. UINT64 keeps its dedicated
DECIMAL(20, 0)representation path.Bolt work needed
References