GH-48476: [C++] [Parquet] Add max_row_group_size writer property to limit row groups by compressed byte size

[zhf999]

Add max_row_group_size writer property to limit row groups by compressed byte size

GH-#48467 Add configure to limit the row group size

PR Description

Rationale for this change

Currently the Parquet writer only supports limiting row groups by row count
(WriterProperties::max_row_group_length()). For tables with wide rows or
highly variable row sizes, a row-count limit alone can produce row groups that
are much larger (or smaller) than desired. Many query engines and storage
systems tune around a target row group size in bytes (e.g. HDFS block size),
so it is useful to also support limiting row groups by their compressed byte
size.

What changes are included in this PR?

• Add WriterProperties::max_row_group_size() and
  WriterProperties::Builder::max_row_group_size(int64_t) to specify the max
  row group size in compressed bytes. The default is unlimited
  (std::numeric_limits<int64_t>::max()), so existing behavior is unchanged
  unless users opt in.

• parquet::arrow::FileWriter::WriteRecordBatch now checks the size of the
  current buffered row group before/after each write, counting:

  • compressed pages already written to the sink
    (RowGroupWriter::total_compressed_bytes_written()),
  • compressed pages still buffered in column writers
    (RowGroupWriter::total_compressed_bytes()),
  • the buffered dictionary not yet written as dictionary pages
    (RowGroupWriter::estimated_buffered_stats().dict_bytes, uncompressed).

  When the accumulated size reaches the limit, a new buffered row group is
  started. The check happens at write boundaries, so a row group may exceed
  the limit by up to the size of a single written batch.

• The limit only applies to buffered row groups (WriteRecordBatch).
  WriteTable and the manual NewRowGroup()/WriteColumnChunk() paths
  control row group boundaries explicitly via chunk_size/API calls and are
  intentionally not affected (row counts are fixed before writing in the
  non-buffered mode, so the byte size limit cannot be enforced there). This is
  documented on the builder method.

Are these changes tested?

Yes.

• WriterPropertiesTest.RoundTripThroughBuilder (properties_test.cc) now
  covers max_row_group_size, with a non-default value added to the
  override_defaults test case.
• New test TestArrowReadWrite.WriteRecordBatchRespectsMaxRowGroupSize
  (arrow_reader_writer_test.cc) verifies that record batches exceeding the
  byte size limit cause row group rollover on subsequent writes.
• Existing WriteRecordBatch/WriteTable tests pass unchanged, confirming no
  behavior change with the default (unlimited) setting.

Are there any user-facing changes?

Yes, a new opt-in writer property max_row_group_size is added to the public
C++ API (WriterProperties / WriterProperties::Builder). The default is
unlimited, so there is no behavior change for existing users.

• GitHub Issue: [C++][Parquet] Add configure to limit the row group size #48467
• GitHub Issue: GH-48475: [Ruby] Add support for reading Int32 and UInt32 arrays #48476

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #48476 has been automatically assigned in GitHub to PR creator.

[zhf999]

Related PR: #48468

[Reranko05]

GitHub Issue: #48476

Thanks for contributing! Can you edit the PR desc and ref to the issue, it seems a PR is being referenced here.

[wgtmac]

Why only for WriteRecordBatch?

[zhf999]

WriteRecordBatch writes into a buffered row group, which accumulates across calls. That makes the byte size observable between writes, so we can simply stop adding to the current row group once the accumulated compressed size reaches the limit.

While WriteTable uses the non-buffered path, where each call maps to exactly one row group whose row count is fixed up front by the user-supplied chunk_size. Enforcing a byte limit there would require predicting the compressed size before writing, e.g. by estimating an average row size from the previous row group. That's inherently approximate, especially since chunk_size is an explicit contract from the caller.

I wonder if estimation is acceptable in WriteTable?

[wgtmac]

IIRC, WriteTable also splits the table into several row groups if max_row_group_length is reached.

[zhf999]

max_row_group_length can be applied there because row counts are known before writing — it just clamps chunk_size. The byte size simply isn't knowable at that point, which is the asymmetry.

[github-actions]

⚠️ GitHub issue #48476 has been automatically assigned in GitHub to PR creator.