Component Selection
Describe the Bug
The Bolt Parquet writer can keep a highly compressible column in one DataPage
for an extremely large number of logical rows.
Before this fix, a page was closed mainly when its estimated encoded value size
reached data_pagesize. This is not a sufficient bound for dictionary-encoded,
low-cardinality, or all-null columns: their encoded value buffers may remain
small while the logical value/row counters continue to grow.
Parquet stores DataPageHeader.num_values and the corresponding DataPageV2
counts as signed 32-bit integers. Once the writer's 64-bit in-memory counter
exceeds INT32_MAX, the unchecked narrowing conversion produces an invalid
page header. The write can still finish successfully, but a conforming reader
later rejects the file.
One production file contained:
ColumnMetaData.num_values = 2,785,305,817
DataPageHeader.num_values = -1,509,661,479
The second value is the signed 32-bit result of the first value modulo
2^32:
2,785,305,817 - 4,294,967,296 = -1,509,661,479
Reproduction Steps
- Write a Parquet column using dictionary encoding and compression.
- Use a low-cardinality
BYTE_ARRAY value, or an optional column containing
mostly/all nulls, so that encoded bytes grow much more slowly than logical
row count.
- Allow a very large row group and repeatedly append batches to the same
column writer.
- Continue until the current DataPage contains more than
INT32_MAX logical
values.
- Close the writer and read the generated file.
The writer can report success, while the reader fails with an error similar to:
Expected -1509661479 values in column chunk ... but got 0 values instead
Bolt Version / Commit ID
main branch @bd7d19c
System Configuration
- **Build Type**: Release
- **CPU Arch**: AMD64
- **Framework**: Spark with Gluten/Bolt
Logs / Stack Trace
Expected -1509661479 values in column chunk at offset 32065246
but got 0 values instead over 0 pages ending at file offset 32065223
Expected Behavior
The writer should:
- Close a DataPage when either its byte-size limit or logical row-count limit
is reached.
- Use a safe default maximum row count per page.
- Validate every count before converting it to a Parquet page-header
int32.
- Fail during writing, instead of silently producing a file that only fails
during a later read, if an unrepresentable page is ever encountered.
Additional context
Apache Arrow added the same independent row limit because a byte-only limit can
produce excessively large pages for highly compressible data:
The proposed default is 20,000 rows per DataPage, matching current Arrow C++,
Arrow Rust, and parquet-java behavior.
Component Selection
Describe the Bug
The Bolt Parquet writer can keep a highly compressible column in one DataPage
for an extremely large number of logical rows.
Before this fix, a page was closed mainly when its estimated encoded value size
reached
data_pagesize. This is not a sufficient bound for dictionary-encoded,low-cardinality, or all-null columns: their encoded value buffers may remain
small while the logical value/row counters continue to grow.
Parquet stores
DataPageHeader.num_valuesand the corresponding DataPageV2counts as signed 32-bit integers. Once the writer's 64-bit in-memory counter
exceeds
INT32_MAX, the unchecked narrowing conversion produces an invalidpage header. The write can still finish successfully, but a conforming reader
later rejects the file.
One production file contained:
The second value is the signed 32-bit result of the first value modulo
2^32:Reproduction Steps
BYTE_ARRAYvalue, or an optional column containingmostly/all nulls, so that encoded bytes grow much more slowly than logical
row count.
column writer.
INT32_MAXlogicalvalues.
The writer can report success, while the reader fails with an error similar to:
Bolt Version / Commit ID
main branch @bd7d19c
System Configuration
Logs / Stack Trace
Expected -1509661479 values in column chunk at offset 32065246 but got 0 values instead over 0 pages ending at file offset 32065223Expected Behavior
The writer should:
is reached.
int32.during a later read, if an unrepresentable page is ever encountered.
Additional context
Apache Arrow added the same independent row limit because a byte-only limit can
produce excessively large pages for highly compressible data:
The proposed default is 20,000 rows per DataPage, matching current Arrow C++,
Arrow Rust, and parquet-java behavior.