Should pip use PEP 691 JSON filename when the URL basename is not a filename?

[2ykwang]

I may be missing some history here, but pip seems to use the download URL basename as the candidate filename for PEP 691 JSON entries.

This is probably niche, since PyPI URLs appear to end with the distribution filename. but PEP 691 defines filename separately from url, and its FAQ notes that it is not clear whether repositories always expose the filename as the last part of the URL. ¹

with an entry like this:

{
  "filename": "simplewheel-1.0-py2.py3-none-any.whl",
  "url": "/download/simplewheel",
  "hashes": {}
}

pip seems to treat the URL basename simplewheel as the filename and skips the entry as not a file before downloading, despite the valid JSON filename. the same wheel installs when the URL ends with the wheel filename.

should pip use the JSON filename field for candidate filename detection in this case, or is relying on the URL basename intentional?

Footnotes

1. PEP 691 - why-include-the-filename-when-the-url-has-it-already) ↩

[pfmoore]

My recollection is that some authenticated indexes provide a synthesised URL that is only valid for a short period. So you could have something like

{
  "filename": "simplewheel-1.0-py2.py3-none-any.whl",
  "url": "/download/temp-names/ab562f6783fa8721",
  "hashes": {}
}

In that case it's clearly not reasonable to use the URL basename. I have a vague recollection that this was brought up on discuss.python.org at some point, but my search skills are failing me right now.

IMO, we should prefer filename if it's present, and only if it isn't should we fall back on the basename of the URL.

[2ykwang]

IMO, we should prefer filename if it's present, and only if it isn't should we fall back on the basename of the URL.

Thanks @pfmoore I agree.

before opening this issue, I tried to find the related PEP 691 discussion but couldn’t find an exact thread about filename vs URL basename.

these are the closest related discussions I found, in case they’re useful for anyone else looking into the history:

• Supporting signed file URLs in the simple package repository API
• PEP 691: JSON-based Simple API for Python Package Indexes
• Backwards-incompatible change to PyPI JSON API

[2ykwang]

IMO, we should prefer filename if it's present, and only if it isn't should we fall back on the basename of the URL.

agreed. though i'm not sure about one case:

• an index sends filename and its url basename is already valid (so it works today)
• filename differs from the basename but isn't a valid distribution filename

preferring filename there could break something that works today. maybe prefer filename only when it's a valid filename, and fall back to the basename otherwise?

[pfmoore]

Actually, it's worth noting that the part of the index spec referring to the HTML form says

The href attribute MUST be a URL that links to the location of the file for download, and the text of the anchor tag MUST match the final path component (the filename) of the URL.

While there's no equivalent wording in the JSON version, it does make me wonder whether it's even allowable for the filename to not be the basename of the URL. The spec simply isn't clear enough here.

But in any case, it doesn't matter. This is a breaking change regardless. If we have

{
  "filename": "simplewheel-1.0-py2.py3-none-any.whl",
  "url": "/download/simplewheel-2.0-py2.py3-none-any.whl",
  "hashes": {}
}

using the filename will change the version number we use.

If we want to be as careful as possible, we should probably use whichever of filename and url has a valid distribution filename. If both are valid, then we should check that they are identical, and reject the entry with an error if they aren't. That could still break existing use (what's now an error would currently silently prefer the url), but I don't think it's a bad thing to flag inconsistent data.

This is now getting sufficiently complex that I'm not sure whether it's worth the effort of changing this. What's the motivation behind the issue? Have actual errors been reported as a result of the current behaviour, or is it simply a matter of doing "the right thing"?

[2ykwang]

thanks @pfmoore. nothing has been reported. I just noticed the mismatch while reading the related code, so this was more about "doing the right thing" than fixing a real bug.

I agree that if the cost outweighs the benefit, it probably isn't worth changing. that said, I'm not sure we can really estimate the cost either. the main thing that could break would be a private index providing an invalid filename, and we don't really know how common that is.

given that, I'd be happy to leave things as they are for now. if this ever seems worth revisiting, I think detecting and flagging a mismatch, as you suggested, would be a better direction than silently preferring one value over the other.

[pfmoore]

Let's see what the other maintainers think. I'm not against fixing it.