Skip to content

Abort should remove incomplete Parquet data files #1247

Description

@fallintoplace

Follow-up from #1241.

That PR changes the failed batch-write path to call Abort() instead of Close(), which is the right control flow for avoiding Parquet footer finalization after a write error. One piece it intentionally leaves alone: ParquetFileWriter.Abort() currently only closes the underlying file handle.

For local files this is mostly just an orphan cleanup concern. For object/blob backends, though, it can leave a footer-less object in storage. Readers should not see it because the snapshot is never committed, but the object can still sit around until orphan cleanup catches it.

Expected behavior:

  • Abort() closes the open writer/file handle.
  • Abort() removes the incomplete file path from the FileIO.
  • If close and remove both fail, both errors are returned, for example with errors.Join.

This also applies to RollingDataWriter's deferred abort path, since it calls the same writer-level Abort() when a stream exits with an open file.

Suggested tests:

  • Abort() removes the created file.
  • remove errors are returned.
  • RollingDataWriter abort removes the incomplete current file, if practical.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions