Skip to content

[Bug] Parquet writer can overflow DataPage row/value counts for highly compressible columns #800

Description

@olemon111

Component Selection

  • Core Engine (Expression eval, Memory, Vector)
  • Connectors / File Formats (Hive, Parquet, etc.)
  • API / Bindings (Python, etc.)
  • Build
  • Other

Describe the Bug

The Bolt Parquet writer can keep a highly compressible column in one DataPage
for an extremely large number of logical rows.

Before this fix, a page was closed mainly when its estimated encoded value size
reached data_pagesize. This is not a sufficient bound for dictionary-encoded,
low-cardinality, or all-null columns: their encoded value buffers may remain
small while the logical value/row counters continue to grow.

Parquet stores DataPageHeader.num_values and the corresponding DataPageV2
counts as signed 32-bit integers. Once the writer's 64-bit in-memory counter
exceeds INT32_MAX, the unchecked narrowing conversion produces an invalid
page header. The write can still finish successfully, but a conforming reader
later rejects the file.

One production file contained:

ColumnMetaData.num_values = 2,785,305,817
DataPageHeader.num_values = -1,509,661,479

The second value is the signed 32-bit result of the first value modulo
2^32:

2,785,305,817 - 4,294,967,296 = -1,509,661,479

Reproduction Steps

  1. Write a Parquet column using dictionary encoding and compression.
  2. Use a low-cardinality BYTE_ARRAY value, or an optional column containing
    mostly/all nulls, so that encoded bytes grow much more slowly than logical
    row count.
  3. Allow a very large row group and repeatedly append batches to the same
    column writer.
  4. Continue until the current DataPage contains more than INT32_MAX logical
    values.
  5. Close the writer and read the generated file.

The writer can report success, while the reader fails with an error similar to:

Expected -1509661479 values in column chunk ... but got 0 values instead

Bolt Version / Commit ID

main branch @bd7d19c

System Configuration

- **Build Type**: Release
- **CPU Arch**: AMD64
- **Framework**: Spark with Gluten/Bolt

Logs / Stack Trace

Expected -1509661479 values in column chunk at offset 32065246
but got 0 values instead over 0 pages ending at file offset 32065223

Expected Behavior

The writer should:

  1. Close a DataPage when either its byte-size limit or logical row-count limit
    is reached.
  2. Use a safe default maximum row count per page.
  3. Validate every count before converting it to a Parquet page-header int32.
  4. Fail during writing, instead of silently producing a file that only fails
    during a later read, if an unrepresentable page is ever encountered.

Additional context

Apache Arrow added the same independent row limit because a byte-only limit can
produce excessively large pages for highly compressible data:

The proposed default is 20,000 rows per DataPage, matching current Arrow C++,
Arrow Rust, and parquet-java behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions