Make the HTTP transport an extra, so `caldav` is installable from PyPI without `niquests`

[clarkewing]

Context: this is a concrete follow-up to #611 ("v4.0 — requests, niquests, httpx or something else?"), and to the packaging concerns raised in #530 and #457. I'm raising it now because #611 says the decision will be revisited before v4.0, and because there is one downstream consumer for whom the current "just patch pyproject.toml" solution does not work.

TL;DR

caldav already supports requests and httpx in code, and the fallback is CI-tested. The only thing preventing Home Assistant from adopting caldav 3.x is a single line of metadata — "niquests" in [project] dependencies.

Unlike Debian or Gentoo, Home Assistant installs from PyPI and cannot patch pyproject.toml. So the workaround offered to distro maintainers is structurally unavailable to what is plausibly one of this library's largest sources of end users.

I'd like to propose moving the HTTP transport into extras.

Why Home Assistant can't do what the distros do

In #530, the resolution was that Gentoo would sed the dependency themselves, and in #611 the position is "it's 'just' to edit the pyproject.toml file." That is entirely reasonable for a distro that builds from source.

Home Assistant doesn't build from source. It declares dependencies in an integration manifest and installs the published wheels from PyPI:

"requirements": ["caldav==2.1.0", "icalendar==6.3.1", "vobject==0.9.9"]

There is no patch step. Whatever caldav's metadata requires, Home Assistant installs. So the escape hatch that resolved #530 for distros does not exist here.

The concrete blocker

caldav>=3.0 requires niquests, which requires urllib3-future, whose wheel ships a top-level urllib3_future.pth. Per Python's site module, any line in a .pth beginning with import is executed at every interpreter startup. That file's single line does, in effect:

shutil.rmtree(site-packages/urllib3)
shutil.copytree(site-packages/urllib3_future, site-packages/urllib3)

Home Assistant's dependency validator (hassfest) forbids top-level .pth files in dependencies. This rule was added in home-assistant/core#166411 as a supply-chain security control, in direct response to a real attack in which a malicious .pth was used to achieve code execution at interpreter startup.

Home Assistant contributors already tried the bump:

• Bump caldav to 3.2.0 home-assistant/core#167460 — "Bump caldav to 3.2.0" fails CI with:
  Package urllib3-future has a forbidden file 'urllib3_future.pth' in niquests

The bump has been stuck ever since. Home Assistant is therefore pinned to caldav==2.1.0.

I also checked the other two ends of this, so as not to push work onto this library unnecessarily:

• urllib3-future will not drop the .pth. Remove urllib3_future.pth jawah/urllib3.future#337 ("Remove urllib3_future.pth") was opened citing this exact Home Assistant CI failure, and was closed as not planned. The README contains a considered rebuttal of the alternatives. The only opt-out (URLLIB3_NO_OVERRIDE=1) is build-time only and requires a source build, which doesn't help a PyPI-wheel consumer.
• Home Assistant owners are very unlikely to grant an exception. An allowlist exists, but every entry in it is a passive .pth (namespace shims, distutils precedence, coverage hooks). urllib3-future's is the only one that rewrites site-packages on every startup — precisely the behavior class the rule was written to catch — and it would replace urllib3 for the entire Home Assistant installation, affecting every other integration.

That leaves caldav's metadata as the only tractable point in the chain. Hence this issue.

Proposal

Support for requests is already built into the code, and CI already runs the test suite with requests in place of niquests. Given that, we could envision moving the HTTP libraries out of [project] dependencies and into extras:

[project]
dependencies = [
  "lxml",
  "recurring-ical-events>=2.0.0",
  "icalendar>6.0.0",
  "icalendar-searcher>=1.0.5,<2",
  "dnspython",
  "python-dateutil",
  "PyYAML",
]

[project.optional-dependencies]
niquests = ["niquests"]   # recommended: HTTP/2/3, sync + async
requests = ["requests"]   # sync only
httpx    = ["httpx"]      # sync + async

with pip install caldav[niquests] documented as the recommended install, so niquests remains the default and preferred transport for everyone who doesn't have a reason to avoid it.

As identified in #530, PEP 621 has no "any-one-of" dependency, and extras can only add — so a bare pip install caldav would install a package that raises ImportError on first use. That is obviously a cost, and it is the crux of the trade-off: a clear, immediate error on a bare install, versus total unavailability for consumers that cannot patch metadata. A few mitigations could soften it:

• Raise a clear, actionable error at import/client-construction: "No HTTP library found. Install one of: pip install caldav[niquests] (recommended), caldav[requests], or caldav[httpx]." async_davclient.py already does something close to this for its niquests → httpxyz → httpx chain.
• Lead the README/docs install section with pip install caldav[niquests].
• Precedent within this project: Make vobject into an optional dependency #527 made vobject an optional dependency, and v2.1.0/v2.1.1 in fact shipped with no HTTP library in base dependencies at all.

If extras aren't the right shape, I'd be glad to explore alternatives — the goal is a PyPI-installable path without niquests, not this specific mechanism.

Why this could be worthwhile

Beyond unblocking the version bump, this opens a door @tobixen pointed at in #457:

"I have an impression that HomeAssistant is one of the main drivers of end users to the caldav project. They already have httpx and aiohttp (and requests) in the dependency list, but not niquests. To get upgrades smoothly into the homeassistant project I suppose httpx is preferred."

Home Assistant is an asyncio application, and it ships httpx along with a helper that hands integrations a shared async httpx client. If Home Assistant could install caldav[httpx], the integration could:

• use caldav's async client instead of trampolining every call through a thread-pool executor — being sync-only forces that trampolining throughout, and it has been a recurring source of bugs (see Cap concurrent CalDAV requests to prevent thread-pool starvation home-assistant/core#172037, "Cap concurrent CalDAV requests to prevent thread-pool starvation"), and
• inject Home Assistant's shared session.

Two of the three requirements for Home Assistant's highest integration quality tier — async-dependency and inject-websession — are unreachable for CalDAV today purely because of the transport, no matter how good the integration code becomes. So this metadata change is the unlock for caldav being a first-class async citizen in what is likely one of its biggest downstream consumers.

Next steps

I'm happy to create a PR with the pyproject.toml change, the friendly ImportError, docs updates, and a CI matrix entry per transport — if there's openness to the direction. I'd rather agree the shape here first than send an unsolicited packaging change.

Thanks for caldav, and for the care you've already shown on the packaging question in #530; this is very much intended as a continuation of that conversation rather than a re-litigation of it.

References

• v4.0 - requests, niquests, httpx or something else? #611 — v4.0 transport decision (open)
• Please restore compatibility with requests, as niquests is not suitable for packaging #530 — "Please restore compatibility with requests, as niquests is not suitable for packaging"
• Replace requests with niquests or httpx? #457 — "Replace requests with niquests or httpx?"
• Make vobject into an optional dependency #527 — "Make vobject into an optional dependency" (precedent)
• Bump caldav to 3.2.0 home-assistant/core#167460 — "Bump caldav to 3.2.0" (blocked by this)
• Warn about *.pth files in dependencies home-assistant/core#166411 — the .pth ban, and its rationale
• Cap concurrent CalDAV requests to prevent thread-pool starvation home-assistant/core#172037 — thread-pool starvation from sync-in-async
• Remove urllib3_future.pth jawah/urllib3.future#337 — "Remove urllib3_future.pth" (closed, not planned)

[tobixen]

It's important to support HomeAssistant. Release 3.3.0 has been expected to be released "within a week" for the last few weeks already, but I'll consider to do something with this issue before releasing 3.3.0.

[tobixen]

I think it's a big problem if things would suddenly break when upgrading from 3.2.1 to 3.3.0 without adding [niquests] in the pyproject.toml. Perhaps it could be to have a fork like caldav-no-httplib on pypi? (or perhaps remove all the code from the caldav library, and have it depend on a caldav-client[niquests] package).

[clarkewing]

Fair point. I think your fork/split options would both work, and I appreciate you thinking about how to make it happen. My hesitation is that either one commits you to maintaining a second distribution indefinitely, and I don't think a fork or a package split is needed to resolve this. The issue comes from leaving the base dependency empty. Extras can only add, so a solution could be to give the floor a boring, universally-installable HTTP library rather than nothing:

[project]
dependencies = [
  "requests",   # default transport: universally packaged, no .pth, no surprises
  ...
]

[project.optional-dependencies]
niquests = ["niquests"]   # HTTP/2 + HTTP/3, sync + async — recommended
httpx    = ["httpx"]      # sync + async

That gives:

• pip install caldav → works out of the box, on requests. No ImportError, no fork, no second distribution to maintain.
• pip install caldav[niquests] → exactly today's behaviour.
• Home Assistant depends on plain caldav → no niquests, no urllib3-future, no .pth, no hassfest failure.

Given pip has no autoremove, it does not uninstall orphaned dependencies on upgrade. So a user upgrading in place from 3.2.1 keeps niquests on disk, and the existing preference chain in davclient.py still selects niquests. Existing installations see no behavioural change at all. Nothing breaks, and nothing silently downgrades.

The only case that changes is a fresh install into a clean environment, which would come up on requests instead of niquests (so HTTP/1.1 rather than HTTP/2). That also concerns an environment rebuilt from a lockfile that pins only caldav. It's a default change, not a break, but it is a real change and would likely belong in the changelog and release notes, with pip install caldav[niquests] recommended prominently in the README.

If however that still feels like too much for a minor release, the same change could slot into 4.0. Which brings me to the next point: there's genuinely no rush on our side. The only thing gated on this issue is converting the integration to the async client. So please optimize for whatever is cleanest for caldav, not for whatever is fastest for us. If that means landing this in 4.0 rather than 3.3.0, that's genuinely fine.

One note on the async side, so it's not a surprise later: async would still require caldav[niquests] or caldav[httpx], since the async chain doesn't include requests. That's fine for us: Home Assistant already ships httpx and provides a shared async client.