Added an API for working with wheel files

[agronholm]

Closes #697.

[agronholm]

Note to self: remember to take into account older versions of specs (pypa/wheel#440 (comment)) when normalizing/validating wheel names.

[henryiii]

Would it make sense to have the other direction, too? A to_filename method? Or even just a __str__ that would produce the filename? I had a method like this:

@property
def name_ver(self) -> str:
    name = self.metadata.canonical_name.replace("-", "_")
    # replace - with _ as a local version separator
    version = str(self.metadata.version).replace("-", "_")
    return f"{name}-{version}"

@property
def basename(self) -> str:
    pyver = ".".join(sorted({t.interpreter for t in self.tags}))
    abi = ".".join(sorted({t.abi for t in self.tags}))
    arch = ".".join(sorted({t.platform for t in self.tags}))
    optbuildver = (
        [self.wheel_metadata.build_tag] if self.wheel_metadata.build_tag else []
    )
    return "-".join([self.name_ver, *optbuildver, pyver, abi, arch])

Edit: ahh, I see the new helper function. But it seems like wrapping it in a method here would be natural. But the functionality is there in make_wheel_filename.

Why isn't this BuildTag | None, by the way? I guess it supports an empty tuple. Though then it seems make_wheel_filename could default to an empty tuple instead of supporting None.

[agronholm]

Why isn't this BuildTag | None, by the way? I guess it supports an empty tuple. Though then it seems make_wheel_filename could default to an empty tuple instead of supporting None.

Can you clarify what you're talking about here?

[henryiii]

How would one support prepare_metadata_for_build_wheel using this? I had functionality to only write out the metadata dir. I guess you could write an empty wheel then copy out the .dist-info directory; is that the only way?

[henryiii]

Why default to 2.1? 2.3 is the current version, and in general, a default can't be safely updated, so IMO this should be required.

[agronholm]

I wasn't aware; I just copied this from the old code.

[agronholm]

How would one support prepare_metadata_for_build_wheel using this? I had functionality to only write out the metadata dir. I guess you could write an empty wheel then copy out the .dist-info directory; is that the only way?

Separating the metadata preparation step should cover that use case. No need to jump through unnecessary hoops then.

[agronholm]

How would one support prepare_metadata_for_build_wheel using this? I had functionality to only write out the metadata dir. I guess you could write an empty wheel then copy out the .dist-info directory; is that the only way?

Separating the metadata preparation step should cover that use case. No need to jump through unnecessary hoops then.

What changes would that actually entail? From what I can see, only write_metadata() would need to be separated into a free function that would take the package base name and the metadata version as its arguments, yes? Anything else?

[MrMino]

Hello there 👋 I'm the person who had the bright idea to build this jankyness back in the day, and now has to maintain it for the sake of all internal APIs we've built with it in the company I work for :).

It would be great to have one method for writing both files and directories. This way, if there's a list of paths to include (e.g. from a configuration file), the code doesn't have to perform the if os.path.isdir(path): dance. It also adds nice symmetry for people who would use this in conjunction with tarfile.TarFile.add to create a PEP-517 builder.

Out of curiosity: what's the reason behind the split between WheelReader and WheelWriter? I.e. why not just have a single class with different modes of operation? I realize that doing it this way makes the code far simpler, but is there any other reason, perhaps related to IO considerations?

[agronholm]

It would be great to have one method for writing both files and directories. This way, if there's a list of paths to include (e.g. from a configuration file), the code doesn't have to perform the if os.path.isdir(path): dance.

Can you elaborate on this? Only write_files_from_directory() uses Path.is_dir(), but I don't see how that's a problem.

It also adds nice symmetry for people who would use this in conjunction with tarfile.TarFile.add to create a PEP-517 builder.

I frankly don't understand what tar files have to do with wheels.

[MrMino]

Can you elaborate on this? Only write_files_from_directory() uses Path.is_dir(), but I don't see how that's a problem.

It is the API user that will have to Path.is_dir() each path, and that's my point.

I would like to do a WheelWriter.write(path) without having to worry whether its a directory or a file. Right now I have to perform a check first, only then I'll know whether I can use write_files_from_directory or just write_file. Or am I missing something here?

Since you are already performing that check in write_files_from_directory, why not make a write method that can take in both files and directories?

I frankly don't understand what tar files have to do with wheels.

PEP-517 defines two hooks: one for bdists, and one for sdists. I can TarFile.add a path to an sdist, without having to check whether it's a dir or not, yet I cannot WheelWriter.write it to a wheel. It's a minor thing, but if someone wants a builder for both sdists and bdists, it makes the code cleaner when whatever paths are chosen to be added can be treated similarly by both hooks.

[agronholm]

I would like to do a WheelWriter.write(path) without having to worry whether its a directory or a file

Can you elaborate on your use case? Why can you not use write_files_from_directory()?

[MrMino]

Why can you not use write_files_from_directory()?

Because it will throw a WheelError if the user input results in a path to a file being given to it, e.g. when adding files from a list of paths. I need to check each path and switch to write_file if that's the case, which is a pattern that the API should cover for me. I'm questioning the ergonomics of having to choose between those two methods at runtime. The API should be liberal in what it accepts without me having to add the is_dir() boilerplate.

I'm unsure what there is to elaborate further. I feel like I'm wasting everyone's time with a relatively minor point.

The code is already making the necessary check in write_files_from_directory, why not just write_file the path there instead of raising the WheelError(f"{basedir} is not a directory")? All that is left afterwards is renaming it to something more generic, e.g. write.

[agronholm]

I'm unsure what there is to elaborate further. I feel like I'm wasting everyone's time with a relatively minor point.

I've been asking about your use case and you haven't been answering my question, so I will keep asking until you do.

What reason do you have to use the API to write individual files/directories to the wheel? That's what I've been wanting to find out. Are the contents coming from outside the file system?

[MrMino]

:) I thought this was good enough: writing to a wheel based on a list of paths. We are talking about generic mechanic, so I gave a generic use case.

This should come down to API design principles, not to whether I'm capable of coming up with examples on the level of specificity you expect, but let me try:

A dead-simple build backend that creates wheels by packaging only the paths given by the user, as they are specified inside a list in one of pyproject.toml tables: ["file.py", "some_src_dir", "some_other_dir/data.bin"].

Speaking personally, I don't want to have to care whether each path is a directory/symlink/file. That's all there is to this. Really. I don't need 2 (or, god forbid, 3) methods for this, and I don't want to have to choose between them based on my own path checks - I'd like the API to take care of the checking and choosing what's right for each path.

What reason do you have to use the API to write individual files/directories to the wheel? That's what I've been wanting to find out. Are the contents coming from outside the file system?

It's akin to asking "what reason do you have to use the TarFile to write individual files/directories to a tarball". Lots! 😉. Maybe we are talking past each other - I still want the directories to be walked through and recursively added to the wheel. I just don't want to have to check and choose different method for each path. It's just one if path.is_dir(): ... else ... clause less for the users of this API. 😄

There's no esoteric use case here. It's just a trivial point about the ergonomics of this API. I don't want to bikeshed this any further.

[agronholm]

My point is that if you're unable to come up with any practical use case for making this change, then why should I do it? It's hardly economical for write() to check if the argument is a file or directory, given the presence of write_files_from_directory(), as ithe former involves an extra system call. For more esoteric use cases, it can also accept a bytestring or unicode string, or even an open file object.

[MrMino]

My point is that if you're unable to come up with any practical use case for making this change, then why should I do it?

I gave you a practical use case. I already use this to help QA folks package their Robot Framework sources into wheels. They do not necessarily have a good understanding of how Python build systems work, so having an explicit list of paths helps a lot.

It's hardly economical for write() to check if the argument is a file or directory, given the presence of write_files_from_directory(), as ithe former involves an extra system call.

If the cost of adding one syscall to write_files_from_directory() is too high, it would be far more productive for me and you if you just led with that. I was honestly second guessing whether we are talking about the same thing or whether I'm understanding the objectives of this API.

I don't think it's a good argument. write_files_from_directory() already does that syscall on L520, it just doesn't do anything useful with that check apart from bouncing the API user if they forget to check if the path is actually a directory.

Further, current semantics mean that the code using this API has to perform this check twice: once to figure out which method to use, then, again inside write_files_from_directory(). So it's actually less economical this way.

Why not just make that method do both and have one less sharp edge?

[brettcannon]

@agronholm what's draft about this PR? Are you looking for feedback on the API first?

[agronholm]

@agronholm what's draft about this PR? Are you looking for feedback on the API first?

Yes, I'd very much like feedback from potential downstream users to ensure that the API is comfortable enough for them to adopt into their own projects.

[pfmoore]

Is there any documentation of the new API? I don't really want to read the implementation to review the API, and as far as I can see what docs will exist in the final build are autogenerated, but there are no docstrings. So if I look at https://packaging--805.org.readthedocs.build/en/805/wheelfile.html, it's empty.

[agronholm]

Is there any documentation of the new API? I don't really want to read the implementation to review the API, and as far as I can see what docs will exist in the final build are autogenerated, but there are no docstrings. So if I look at https://packaging--805.org.readthedocs.build/en/805/wheelfile.html, it's empty.

Fair enough, I suppose. Then it would be sufficient at this juncture to add some docstrings and then have downstream project reps look at it?

[henryiii]

I tried asking Fable to swap out the custom wheel writer in scikit-build-core with this PR, and report on the challenges, missing parts, etc. Report below.

(Just copying what it told me below, rather than having it report here)

I could take over the PR, rebase on main, and try to add the stuff that's missing to make it usable. I could also do the exact same experiment (since it didn't actually require knolege of scikit-build-core's internals by me) on a few other backends and see what's missing for those, too. @agronholm, let me know how you'd like me to proceed. I can do as little or as much as needed.

(And yes, I mistyped the PR number as 804, it didn't care though)

🤖 AI text below 🤖

Done — the experiment lives on branch try-packaging-wheelfile (one commit, 48e42da), with scikit-build-core's WheelWriter internals fully delegated to packaging.wheelfile from your PR checkout (note: it's actually PR #805; 804 is the adjacent Emscripten-tags PR). The full non-isolated test suite passes — 990 tests plus all 38 compile-marked builds — except four tests that fail on a genuine API gap (permissions). Wheels built through it install fine with pip and uv, entry points work, and builds are bit-for-bit reproducible across runs. Here's the feedback material, roughly in priority order.

Real bugs found

1. The .dist-info directory name violates the spec for any name that isn't already a single word. WheelMetadata.name is a NormalizedName (hyphens), and the writer builds f"{name}-{version}.dist-info" from it directly, so my_pkg-1.0-py3-none-any.whl contains my-pkg-1.0.dist-info instead of the escaped my_pkg-1.0.dist-info. Real-world impact confirmed: pip cannot install the resulting wheel (it mis-splits the dir name and dies with InvalidVersion: 'pkg2-1.0'); uv happens to tolerate it. The reader has the mirror-image bug: it first probes {normalized-name}-{version}.dist-info, so for spec-correct wheels of multi-word packages the fast path never matches, and the legacy-scan fallback then assigns the raw directory prefix (hyph_pkg) to self.name, silently breaking the NormalizedName contract. I worked around it by passing an already-escaped name as metadata.name.

2. write_file records the wrong size for a stream not at position 0. It uses contents.tell() as the RECORD size: writing an IO[bytes] that had 5 of 10 bytes left produced a RECORD entry claiming size 10 for a 5-byte member. It should count bytes actually written.

3. Timestamp edge cases surface as cryptic errors from inside zipfile. A pre-1980 timestamp raises ValueError: ZIP does not support timestamps before 1980; a post-2107 one raises 'H' format requires 0 <= number <= 65535. Clamping (or a clear early error) would be friendlier — SOURCE_DATE_EPOCH=0 is common in the wild. Related: write_file converts via time.gmtime(timestamp.timestamp()), which breaks past 2038 where time_t is 32-bit (scikit-build-core hit this in the wild and deliberately avoids time.gmtime). Also, a naive datetime gets interpreted in local time via .timestamp() — worth documenting that UTC is expected.

Missing for scikit-build-core to adopt it

• No permission control at all. This is the one gap the test suite itself caught (4 failures). write_file copies st_mode from disk for path/stream input and hardcodes 0o664 for bytes/str, with no override. Reproducible builds need to normalize permissions (0o644 / 0o755, the distlib/wheel convention), and there's no way to do it short of reaching into _zip. A mode= parameter (or a normalize-permissions option on the writer) would cover it.
• Bytes/str content gets no S_IFREG file-type bits in external_attr (path/stream input gets them via S_IFMT). Inconsistent, and some tools care.
• No way to write just the .dist-info tree for prepare_metadata_for_build_wheel — you already raised this on the PR; the experiment confirms it: I had to keep scikit-build-core's entire parallel dist_info_contents() machinery, so METADATA/WHEEL generation stayed duplicated rather than delegated.
• Filename construction can't be fully delegated. make_wheel_filename handles build tags but not variant labels (wheel-variants experiment), so I compute the filename myself and must pass explicit WheelMetadata (the filename won't parse). Not asking the PR to support variants, but it's a data point that the writer being strict about parsing its own filename limits forward-compatibility; explicit-metadata mode handles it, which is good.
• Auto-written RECORD/WHEEL always get the fixed 1980 timestamp, so in a non-reproducible build they diverge from the rest of the archive. A writer-level default timestamp would fix that.
• Build tag ergonomics: settings-level build tags are strings ("7abc"); I had to regex-parse into BuildTag myself. A public parse helper (the regex already exists privately in utils) or str acceptance would help.

Smaller notes

• WheelReader refuses a valid wheel whose file name isn't parseable (e.g. copied to repro-1.whl) even though the content identifies itself; passing a file object works, but accepting a strict/name-optional path mode would be nice for temp files.
• extractall doesn't restore permission bits (executables lose +x) — either restore from external_attr or document it.
• write_distinfo_file uses filename.strip() where write_data_file uses filename.strip("/") — looks like a typo, though PurePath normalization happens to hide it.
• No docstrings/docs yet (pfmoore already raised this).

What works well

• The adapter was genuinely pleasant to write — the delegation removed scikit-build-core's RECORD/hashing/zip bookkeeping (csv, hashlib, base64, zipfile imports all gone), and hashing-while-writing is strictly better than our current write-then-re-read-the-whole-zip RECORD pass.
• Context-manager auto-finalization (RECORD always, WHEEL only if you didn't write your own) composed perfectly with an existing backend that generates its own WHEEL.
• write_file's polymorphic contents (bytes/str/path/stream) mapped 1:1 onto our write/writestr split, and explicit-WheelMetadata mode is exactly what a backend with its own filename logic needs.
• Round-trip works: built wheels pass WheelReader.validate_record(), and everything installed cleanly with pip and uv once the naming bug was worked around.

To try it: git switch try-packaging-wheelfile && uv sync (packaging is wired to your local checkout via tool.uv.sources). The gaps are marked with GAP (packaging#805) / WORKAROUND (packaging#805) comments in src/scikit_build_core/build/_wheelfile.py, and my standalone repro script is at the scratchpad's probe_wheelfile.py if you want to paste minimal repros into the PR. Isolated-marker tests will fail on this branch since released packaging lacks wheelfile — expected. Want me to draft this up as a PR comment for #805?

[agronholm]

I'm okay with you taking over the PR if you want @henryiii

[henryiii]

I did a direct rebase. I ran an adversarial review with Fable - which downgraded to Opus almost immediately. I guess wheels are dangerous? :P

Anyway, I'll work on a combination of this and the above message over the next few days.

• Stage 1: rebase
• Stage 2: fix up some basic reviews
• Stage 3: TBD

🤖 AI text below 🤖

🤖 _Opus adversarial review_ 🤖

I broke it in several ways. Here's the adversarial review of the wheelfile feature.

Critical — path traversal (zip-slip) in WheelReader.extractall

wheelfile.py:319-328 joins archive member names straight onto the destination with no containment check:

target_path = basedir.joinpath(fname)

A malicious wheel escapes the target directory. I built one with a ../evil.txt member (and matching RECORD hash) and extractall wrote it to the parent of the destination. Absolute member names are worse — Path("/dest").joinpath("/etc/whatever") returns /etc/whatever, so an absolute path member writes anywhere the process can. For a foundational packaging library this is the serious one. The fix is to resolve each target and confirm it stays under basedir before opening it for write.

Correctness — make_wheel_filename never validates or normalizes the version

utils.py:180-198. A commit on this branch deliberately dropped the canonicalize_version(version) call to preserve trailing zeros, but nothing replaced it, so any string goes in verbatim:

• make_wheel_filename("foo", "1.0+local-1", tags) → foo-1.0+local-1-py3-none-any.whl, which parse_wheel_filename reads back as version 1.0+local — the -1 is silently lost. Round-trip is broken.
• make_wheel_filename("foo", "not a version", tags) → 'foo-not a version-py3-none-any.whl', a filename with spaces.

It accepts str | Version; at minimum it should coerce a str through Version() (rejecting junk) while still stringifying without stripping zeros. As written, the trailing-zero fix traded one bug for a larger one.

Correctness — streaming hash check has an off-by-one, so a tampered file can read clean

WheelArchiveFile.read, wheelfile.py:117: verification only fires when amount < 0 or len(data) < amount. If a caller reads exactly filesize bytes in one read(n) call, the returned chunk equals amount, the check is skipped, and they get tampered bytes with no exception — the check only triggers on a subsequent read returning short. The internal callers (read_file uses read(-1); validate_record/extractall loop to empty) are safe, but the whole point of this class is verify-on-read, and a user calling f.read(known_size) once is unprotected. Track completion by comparing _num_bytes_read against filesize rather than by chunk length.

Robustness — hostile RECORD raises the wrong exception type

_read_record, wheelfile.py:242-244. A RECORD row with the wrong column count raises ValueError: not enough values to unpack, and a hash field with a stray = (sha256=ab=cd) raises ValueError: too many values to unpack — both leak out instead of a clean WheelError. Everything else in the reader wraps failures in WheelError; these two paths don't.

Footgun — iterate_contents closes the stream when the iterator advances

wheelfile.py:292-297 yields the stream from inside a with self._zip.open(...), so the stream is closed the moment the caller advances the generator. list(wr.iterate_contents())[0].stream.read() raises ValueError: read from closed file. It only works if you read inside the loop body, which the one test happens to do. Either don't hand out a stream whose lifetime is the iteration step, or document it loudly.

Cleanliness

• Dead code: _dist_name_re (utils.py:74) is defined and never used — it was the intended filename escaper, but make_wheel_filename uses canonicalize_name(...).replace("-", "_") instead. Drop it.
• Inconsistent stripping: write_distinfo_file uses filename.strip() (whitespace) while write_data_file and the read_* helpers use .strip("/"). A leading slash slips through in the dist-info path. Looks like a typo for .strip("/").
• No docs, no docstrings: the module has exactly one docstring (validate_record), make_wheel_filename has none, and docs/wheelfile.rst is a title with currentmodule and no automodule/autofunction — it renders nothing. CLAUDE.md says user-visible features require prose docs; this is a whole new public module with an empty page.
• Redundant: make_wheel_filename does canonicalize_name(name).replace("-", "_").lower() — canonicalize_name already lowercases, so .lower() is dead.

What's fine

The core integrity path works: validate_record correctly catches a genuine hash mismatch (positive control passed), and the __exit__/lint fixes from the rebase are sound. The bones are good; the extraction routine and the filename builder are where it breaks.

Want me to open fixes? I'd prioritize the zip-slip guard and the make_wheel_filename version validation as their own commits with regression tests first, and fold the RECORD-parsing and dead-code cleanups into a third.

Fixes:

7f0b424 — fix(utils): validate version in make_wheel_filename
The version string was interpolated verbatim. Now a str is coerced through Version(), which validates it (junk raises InvalidVersion) and normalizes local separators so "1.0+local-1" no longer round-trips to a different version, while Version.__str__ keeps trailing zeros. Dropped the redundant .lower(). Tests cover trailing-zero preservation, local-separator round-trip, and invalid-version rejection.

0cb16b6 — fix(wheelfile): reject path traversal in extractall
The zip-slip hole. extractall now resolves each target and confirms it stays under the destination via is_relative_to, so ../evil.txt and absolute-path members raise WheelError instead of writing outside. Parametrized test for ../ and nested-../ escapes, asserting nothing lands in the parent dir.

554ce93 — fix(wheelfile): raise WheelError on malformed RECORD rows
Wrong column count and malformed hash fields (sha256=ab=cd) now raise WheelError instead of leaking a raw ValueError. Folded in the two cleanups: removed the dead _dist_name_re regex and switched write_distinfo_file to .strip("/") to match the data/record helpers.

Two things I flagged in the review but left alone, since you picked these three:

• Streaming hash off-by-one in WheelArchiveFile.read — a single read(filesize) skips verification. Internal callers are safe; a direct user reading one exact-size chunk isn't. Worth a follow-up (track _num_bytes_read == filesize instead of chunk length).
• iterate_contents stream lifetime — streams close when the iterator advances, so collecting elements before reading breaks. Needs either a redesign or a loud docstring.

Want me to take on either of those, or open the empty docs/wheelfile.rst docs next? Note these four commits sit on top of the earlier rebase, so the eventual push to update the branch will be a force-push.