[BUG] ParquetDatasetWriter(max_file_size=...) leaks file descriptors: 126k FDs for 1000 partitions

[VibhuJawa]

DRAFT for upstream rapidsai/cudf. Re-verified on 2026-07-30 against cuDF 26.08.00a990, H100 80GB, writing to node-local ext3 and WekaFS (identical result on both). Open before filing: not yet checked against a released version — see Environment.

Describe the bug

ParquetDatasetWriter with max_file_size set exhausts the process file-descriptor limit and dies, on a workload with only 1,000 partitions and a 131,072 FD limit.

A single write_table() call over 1,000 partitions leaves 126,038 descriptors open — about 126 per partition — and they are never released. The second write_table() then fails immediately:

append 0 OK -> fds=126038
RuntimeError: Unable to open file. Linux system/library function call error at:
/__w/kvikio/kvikio/cpp/src/file_utils.cpp:160: Too many open files

!! FAILED during append index 1
!! open fds at failure = 126038,  RLIMIT_NOFILE soft=131072 hard=131072

Traceback:

  File ".../cudf/io/parquet.py", line 2277, in write_table
    self._chunked_writers[-1][0].write_table(grouped_df, part_info)
  File ".../cudf/io/parquet.py", line 1857, in write_table
    self._initialize_chunked_state(
  File ".../cudf/io/parquet.py", line 1962, in _initialize_chunked_state
    self.writer = plc.io.parquet.ChunkedParquetWriter.from_options(options)
  File "pylibcudf/io/parquet.pyx", line 771, in pylibcudf.io.parquet.ChunkedParquetWriter.from_options
RuntimeError: Unable to open file. ... Too many open files

Descriptors accumulate across file rotations rather than being released when a chunked writer is retired. Without max_file_size the same workload completes, so the leak is specific to the rotation path.

Steps/Code to reproduce bug

import os, resource, shutil, sys
import cudf, cupy as cp, pylibcudf as plc
from cudf.io.parquet import ParquetDatasetWriter

ROWS, DIM, CENTROIDS, APPENDS = 500_000, 1024, 1000, 4
OUT = "/tmp/pdw_fd_leak"
USE_MAX_FILE_SIZE = True          # flip to False for the control run
shutil.rmtree(OUT, ignore_errors=True)

def fds():
    return len(os.listdir(f"/proc/{os.getpid()}/fd"))

def make_df(seed):
    cp.random.seed(seed)
    arr = cp.random.standard_normal((ROWS, DIM), dtype=cp.float32)
    return cudf.DataFrame({
        "embedding": cudf.Series.from_pylibcudf(plc.Column.from_cuda_array_interface(arr)),
        "centroid": cudf.Series((cp.arange(ROWS) % CENTROIDS).astype("int32")),
    })

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f"cudf {cudf.__version__}  RLIMIT_NOFILE soft={soft} hard={hard}  fds at start={fds()}")

kw = dict(max_file_size="20MB", file_name_prefix="part") if USE_MAX_FILE_SIZE else {}
w = ParquetDatasetWriter(OUT, partition_cols=["centroid"], index=False, **kw)
for i in range(APPENDS):
    w.write_table(make_df(i))
    print(f"  append {i} OK -> fds={fds()}", flush=True)
w.close()
print(f"  close OK -> fds={fds()}, files={sum(len(f) for _, _, f in os.walk(OUT))}")

With USE_MAX_FILE_SIZE = True:

cudf 26.08.00a990  RLIMIT_NOFILE soft=131072 hard=131072  fds at start=5
  append 0 OK -> fds=126038
RuntimeError: Unable to open file. ... Too many open files   (during append 1)

With USE_MAX_FILE_SIZE = False the run completes and descriptor use is flat:

  append 0 OK -> fds=2038
  append 1 OK -> fds=2038
  append 2 OK -> fds=2038
  append 3 OK -> fds=2038
  close OK -> fds=2038, files=1000

Note that even the control path retains 2,038 descriptors — roughly 2 per partition — after close(). That does not fail, but it suggests descriptors are not released on writer teardown in either path.

Expected behavior

File descriptors should be released as chunked writers are rotated out. A 1,000-partition dataset should need on the order of 1,000 concurrent descriptors at worst, not 126,000 — and close() should return them.

Environment overview

• Environment location: bare-metal, single node of a SLURM cluster
• Method of cuDF install: pip — cudf-cu12==26.8.0a990 (nightly wheel)

Environment details

• NVIDIA H100 80GB HBM3, CUDA 12.9, Python 3.12, libkvikio-cu12==26.8.0a36
• RLIMIT_NOFILE soft = hard = 131,072
• Reproduced writing to node-local ext3 and WekaFS, with identical descriptor counts — not filesystem-specific
• Not yet reproduced on a released version. We attempted 25.10.00 but that environment could not run the reproducer for an unrelated reason (missing libcurand), so we make no claim about which releases are affected.

Additional context

max_file_size is exactly the knob you reach for to stop a small-file explosion when writing many partitions across many appends, and it is the knob that fails. This is a plausible practical reason a 40 TB / 100+ GPU production job found the ParquetDatasetWriter path unusable, though we have not reproduced that job end to end.

Related: rapidsai/cudf#23378 — ParquetDatasetWriter max_file_size overestimates list columns and creates too many files. Same option, same workload; that issue is about how many files get created, this one about the writer running out of descriptors while creating them. They may share a root cause in the rotation logic.

Related drafts from us (this fork, to be filed upstream alongside): #7 (partitioned write memory and throughput — the same workload without max_file_size).