Extend regression test on self-referential extras

[Trenza1ore]

Context

After #14157, self-referential extras is now documented as an official feature (supported unofficially since v21.2), #14157 also introduced two simple test cases into tests/functional/test_install_extras.py:

• test_install_self_referential_extras[separate-specifiers]
• test_install_self_referential_extras[combined-specifier]

The test cases were quite simple - in fact they pass with v20.3 - v21.1.3.

Proposal

Add 9 more test cases to cover expected behaviour and prevent regression.

Should pass with 20.3+

1. nested — nested self-referential extras should work.
2. external — a self-referential extra can mix pkg[a] with an unrelated third-party dep in the same extra.
3. partial — after pkg[a] is already installed, installing pkg[all] with the same version adds only the missing deps.
4. unknown — a self-reference with unknown extra (pkg[missing]) must raise a warning as usual.
5. upgrade-dep — a self-referential extra whose dependency version requirements change across versions (from requiring dep==1 to requiring dep==2) must bump dep to version 2.
6. circular — mutually dependent extras (extra a depends on extra b and vice versa) must resolve normally (pip install pkg[a] installs dependencies from both extras).

Should pass with 21.2+

7 & 8. upgrade-different-extras (two scenarios: had-no-extra or had-different-extras) — with an older pkg already installed (with no extras defined or with different extras defined and installed), upgrading to a new version with an all extra should: 1) pull new deps and 2) downgrade an existing dep.

Should pass with 24.1+

9. normalize — PEP 685 normalization should allow "all": ["pkg[x-y]"] to install extra defined as x_y in pyproject.toml.

Local Test Results

noxfile_self_ref_extras.py

"""Temporary experiment: run self-referential extras tests across historical pip.

Runs the real tests from tests/functional/test_install_extras.py against each
pip version by swapping the pip installed into the test virtualenv template.

Usage:
    nox -f noxfile_self_ref_extras.py
    nox -f noxfile_self_ref_extras.py -- 21.2 26.1.2
"""

import json
import os
import shutil
import tempfile
import urllib.request
from pathlib import Path

import nox
from packaging.version import InvalidVersion, Version

ROOT = Path(__file__).resolve().parent
MIN_VERSION = Version("20.2")
# Old pip needs pkgutil.ImpImporter (removed in 3.12).
PYTHON = "3.11"
# PEP 685 extra normalization in the resolver landed in pip 24.1.
PEP685_MIN = Version("24.1")
NORM_TEST = "test_install_self_referential_extras_name_normalization"

nox.options.sessions = ["self_ref_extras"]
nox.options.reuse_existing_virtualenvs = True

# Pytest plugin loaded via -p: replace HEAD pip in the test venv with a
# historical release. Kept inline so this experiment is a single file.
# Network is blocked under pytest (pytest-subket); wheels are pre-downloaded
# by the nox session into SELF_REF_EXTRAS_WHEELHOUSE.
_PLUGIN = r"""
import os
import shutil
import subprocess
import sys
from pathlib import Path

import pytest


@pytest.fixture(scope="session", autouse=True)
def _swap_historical_pip(virtualenv_template):
    version = os.environ["SELF_REF_EXTRAS_PIP_VERSION"]
    wheelhouse = os.environ["SELF_REF_EXTRAS_WHEELHOUSE"]
    site = Path(virtualenv_template.site)

    for path in list(site.glob("*pip*.pth")) + list(site.glob("__editable__*pip*")):
        if path.is_dir():
            shutil.rmtree(path)
        else:
            path.unlink(missing_ok=True)
    for path in site.glob("pip-*.dist-info"):
        shutil.rmtree(path)
    pip_pkg = site / "pip"
    if pip_pkg.exists():
        shutil.rmtree(pip_pkg)

    subprocess.check_call(
        [
            sys.executable,
            "-m",
            "pip",
            "install",
            "--disable-pip-version-check",
            "--no-cache-dir",
            "--no-index",
            "--find-links",
            wheelhouse,
            f"pip=={version}",
            "-t",
            str(site),
        ],
        stdout=subprocess.DEVNULL,
    )
"""


def _stable_version(version: str) -> Version | None:
    try:
        parsed = Version(version)
    except InvalidVersion:
        return None
    if parsed.is_prerelease or parsed.is_devrelease:
        return None
    return parsed


def _fetch_pip_versions() -> list[str]:
    with urllib.request.urlopen("https://pypi.org/pypi/pip/json") as response:
        data = json.load(response)
    versions: list[str] = []
    for version in data["releases"]:
        parsed = _stable_version(version)
        if parsed is not None and parsed >= MIN_VERSION:
            versions.append(version)
    return sorted(versions, key=Version)


def _resolve_versions(posargs: list[str]) -> list[str]:
    all_versions = _fetch_pip_versions()
    if not posargs:
        return all_versions
    unknown = [arg for arg in posargs if arg.startswith("-")]
    if unknown:
        raise SystemExit(f"unknown flag(s): {unknown}")
    available = set(all_versions)
    missing = [v for v in posargs if v not in available]
    if missing:
        raise SystemExit(f"pip version(s) not on PyPI (stable >=20.2): {missing}")
    return sorted(posargs, key=Version)


def _discover_self_ref_tests(session: nox.Session) -> list[str]:
    """Import the real test module inside the session venv and list tests."""
    out = session.run(
        "python",
        "-c",
        """\
import importlib
mod = importlib.import_module("tests.functional.test_install_extras")
names = sorted(
    name
    for name, obj in vars(mod).items()
    if name.startswith("test_") and "self_referential" in name and callable(obj)
)
assert names, "no self-referential extras tests found"
assert "test_install_self_referential_extras_name_normalization" in names
print("\\n".join(names))
""",
        silent=True,
    )
    assert isinstance(out, str)
    return [line for line in out.splitlines() if line.strip()]


def _ensure_common_wheels(session: nox.Session) -> None:
    wheels = ROOT / "tests" / "data" / "common_wheels"
    if wheels.exists() and any(wheels.iterdir()):
        return
    wheels.mkdir(parents=True, exist_ok=True)
    session.run(
        "python",
        "tools/protected_pip.py",
        "wheel",
        "-w",
        str(wheels),
        "--group",
        "test-common-wheels",
        external=True,
    )


@nox.session(python=PYTHON)
def self_ref_extras(session: nox.Session) -> None:
    """Run test_install_extras self-ref tests against historical pip versions."""
    _ensure_common_wheels(session)

    # HEAD pip in the nox/pytest process (test helpers import pip._*); the
    # plugin replaces only the pip inside each test virtualenv.
    session.install("-e", ".", "--group", "test", silent=True)

    test_names = _discover_self_ref_tests(session)
    session.log(f"From test_install_extras.py: {', '.join(test_names)}")

    versions = _resolve_versions(list(session.posargs))
    session.log(f"Testing {len(versions)} pip version(s)")

    plugin_dir = Path(tempfile.mkdtemp(prefix="self-ref-plugin-"))
    wheelhouse = Path(tempfile.mkdtemp(prefix="self-ref-wheels-"))
    try:
        (plugin_dir / "self_ref_hist_pip.py").write_text(_PLUGIN, encoding="utf-8")

        failures: list[tuple[str, str]] = []
        for version in versions:
            session.log(f"=== pip=={version} ===")
            # Download outside pytest so pytest-subket does not block PyPI.
            session.run(
                "python",
                "-m",
                "pip",
                "download",
                "--disable-pip-version-check",
                "--no-cache-dir",
                "-d",
                str(wheelhouse),
                f"pip=={version}",
                silent=True,
            )
            args = [
                "pytest",
                "tests/functional/test_install_extras.py",
                "-k",
                "self_referential",
                "-p",
                "self_ref_hist_pip",
                "-v",
                "--tb=line",
            ]
            if Version(version) < PEP685_MIN:
                args.append(
                    "--deselect="
                    f"tests/functional/test_install_extras.py::{NORM_TEST}"
                )

            env = {
                "SELF_REF_EXTRAS_PIP_VERSION": version,
                "SELF_REF_EXTRAS_WHEELHOUSE": str(wheelhouse),
                "PYTHONPATH": os.pathsep.join(
                    [str(plugin_dir), os.environ.get("PYTHONPATH", "")]
                ),
                "LC_CTYPE": "en_US.UTF-8",
            }
            try:
                session.run(*args, env=env)
                session.log(f"PASS pip=={version}")
            except Exception as exc:
                failures.append((version, str(exc)))
                session.log(f"FAIL pip=={version}")

        session.log("--- summary ---")
        failed_set = {v for v, _ in failures}
        for version in versions:
            session.log(f"{'FAIL' if version in failed_set else 'PASS'}  pip=={version}")
        if failures:
            session.error(f"{len(failures)} version(s) failed: {', '.join(failed_set)}")
    finally:
        shutil.rmtree(plugin_dir, ignore_errors=True)
        shutil.rmtree(wheelhouse, ignore_errors=True)

20.2 – 21.1.3 (before full support)

pip	separate	combined	nested	external	partial	upgrade-different-extras[had-no-extra]	upgrade-different-extras[had-different-extras]	upgrade-dep	circular	unknown	normalize
20.2	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (missing warn)	—
20.2.1	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (missing warn)	—
20.2.2	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (missing warn)	—
20.2.3	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (missing warn)	—
20.2.4	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (deps missing)	AssertionError (missing warn)	—
20.3	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
20.3.1	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
20.3.2	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
20.3.3	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
20.3.4	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
21.0	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
21.0.1	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
21.1	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
21.1.1	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
21.1.2	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—
21.1.3	pass	pass	pass	pass	pass	AssertionError (older extra warn)	AssertionError (older extra warn)	pass	pass	pass	—

21.2 - 26.1.2

pip	separate	combined	nested	external	partial	upgrade-different-extras[had-no-extra]	upgrade-different-extras[had-different-extras]	upgrade-dep	circular	unknown	normalize
21.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
21.2.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
21.2.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
21.2.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
21.2.4	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
21.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
21.3.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.0	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.0.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.0.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.0.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.0.4	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.1.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.1.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.2.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.2.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
22.3.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.0	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.0.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.1.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.1.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.2.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.3.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
23.3.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
24.0	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	—
24.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
24.1.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
24.1.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
24.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
24.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
24.3.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
25.0	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
25.0.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
25.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
25.1.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
25.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
25.3	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
26.0	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
26.0.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
26.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
26.1.1	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass
26.1.2	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass

[pfmoore]

Seriously, do we need this wall of text to say nothing more than "add some extra tests for self-referential extras"? Do we even need an issue here?

You skipped the issue template, so there's no declaration of whether you used AI to generate this, but please be mindful that it takes maintainer time to read all of this. Brevity is a virtue, and it's one that AI tools are terrifically bad at 🙁

[Trenza1ore]

Seriously, do we need this wall of text to say nothing more than "add some extra tests for self-referential extras"? Do we even need an issue here?

You skipped the issue template, so there's no declaration of whether you used AI to generate this, but please be mindful that it takes maintainer time to read all of this. Brevity is a virtue, and it's one that AI tools are terrifically bad at 🙁

Sorry for the inconvenience added. I skipped the issue template because I didn't know whether this falls into feature or bug. And about the amount of text, I wanted to make sure I provided all information needed so it's less back-and-forth, sorry about that, will avoid this in the future.