[RFC 0180] Remove broken packages or unmaintained leaf packages

[Mic92]

Rendered

[dotlambda]

There should be a rule to notify e.g. everyone who touched the package expression in the past before removal.

[Frontear]

There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?

[Mic92]

There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?

I am sure that someone wants to become a maintainer of a package like this, once we actually become serious about removing it, because their hardware won't function without it. However we leave the decision what to do in this case up to the person merging the pull request i.e. waiting for a reasonable amount of time depending on the importance package.

[AndersonTorres]

There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?

There are some of those packages that are very critical. Ideally they should be kept by a team.

[Mic92]

There are some packages with no maintainer yet don't exactly "become outdated" (an example is the intel-microcode package). How are we going to handle those?

There are some of those packages that are very critical. Ideally they should be kept by a team.

I think having this RFC in place, will make actually possible to find instances like this and hopefully improve the maintenance of these packages. I could see the security team for example maintaining microcode package.

[AndersonTorres]

I could see the security team for example maintaining microcode package.

Or the nixos-hardware team (when they become a NixOS team :D )

[Aleksanaa]

I hope the problem infrastructure or something like it can be implemented first, so we can add warnings to end users like "foo is deprecated and is subject to removal because ...", and guide them to give feedback. If they give reasonable reasons, we will cancel or postpone the removal.

We can divide the deprecation of a package into three stages:

1. Throws a warning to end users but still builds on hydra
2. Make a mark similar to insecure. If the user needs to use it, it needs to be manually allowed and it will not be built on hydra.
3. Completely remove related expressions.

Each of these stages can last three to six months, giving everyone plenty of time to react and plenty of reason to remove it.

[Frontear]

Additionally going back to maintainerless packages, I think itd be best to do a broad sweep and find which packages are still orphans and find them a team, especially for critical packages like the microcode package. Once this is done anything left behind would be non essential stuff, which can follow the removal procedure

[8573]

Each of these stages can last three to six months, giving everyone plenty of time to react

Wouldn't anything shorter than six months per stage not be enough time for users of stable channels to notice before the removal process may move on to the next stage?

[AndersonTorres]

About orphaning, we also need to deal with the silent retirement. There are a truckload of maintainers that did not maintain their codes from a long span of time.

I suggested to do a soft deletion of silent retired maintainers at NixOS/nixpkgs#310759.

Regardless the above:

We have the infamous Zero Hydra Failures event. We can do something similar around the months 3 and 9 in order to mark and sweep unmaintained packages.

[Mic92]

I hope the problem infrastructure or something like it can be implemented first, so we can add warnings to end users like "foo is deprecated and is subject to removal because ...", and guide them to give feedback. If they give reasonable reasons, we will cancel or postpone the removal.

We can divide the deprecation of a package into three stages:

1. Throws a warning to end users but still builds on hydra

2. Make a mark similar to insecure. If the user needs to use it, it needs to be manually allowed and it will not be built on hydra.

3. Completely remove related expressions.

Each of these stages can last three to six months, giving everyone plenty of time to react and plenty of reason to remove it.

I don't see really enough benefit in step 2, it just creates more work for us. Removing packages should not be more work than adding packages.

[balsoft]

While trying to implement the automatic removal of unmaintained leaf packages, we (@balsoft and @infinisil) are considering various approaches. They all require evaluating all packages and then checking whether each is depended on by another one and then using that info to influence something like meta.hasNoMaintainersButDependents which avoids a deprecation warning for maintainerless packages.

1. Evaluate it on a per-PR basis: Add a CI check that requires that meta.hasNoMaintainersButDependents is up-to-date for every PR, meaning the contributors have to add it themselves. Disadvantage: Expensive computationally or hard to achieve (see below)
2. Evaluate it on a weekly basis:
   a. Use ast-grep to add meta.hasNoMaintainersButDependents. Disadvantage: Unreliable due to the nature of ast-grep.
   b. Limit it to pkgs/by-name, have CI create a pkgs/by-name/he/hello/no-maintainers-but-dependents signaling file for relevant packages, then:
   1. Introduce a pkgdir = ./. argument to mkDerivation along with a CI check that ensures it's set to the package directory for all packages in pkgs/by-name. Then mkDerivation can automatically add meta.hasNoMaintainersButDependents if the signaling file exists. Disadvantage: Big change, unclear if worth it
   2. Use an .override/extendDerivation in by-name-overlay.nix to add meta.hasNoMaintainersButDependents after it was callPackage'd. Disadvantage: Slows down every evaluation
   3. Use lib.warn in by-name-overlay.nix if the signaling file exists. Disadvantage: Unable to use standard meta problem control mechanisms.

The cleanest, most universal approach seems to be the first one. In addition to other benefits, it requires a manual intervention (adding the necessary meta flag) by persons interested in the package if they wish to merge their PRs, which incentivises maintainership. However, we've stumbled on an issue with Nix, namely that it's difficult to extract dependency information from derivations efficiently. This makes any automated checks for whether a package is leaf time-expensive, in an environment where we're burning a lot of CI minutes already.

In particular:

• Nixpkgs CI currently uses nix-env -qa for evaluation. It is quite fast because nix-env uses readOnlyMode and avoids writing any derivations to disk. However, nix-env does not have a capability to print input derivations, mostly because the underlying PackageInfo structure doesn't contain it. It is not possible to naively patch Nix to pass through the inputDrvs from the Derivation structure to PackageInfo, because derivationStrictInternal actually discards that Derivation after writing the drv file to disk.
  • If there was a way to access the inputDrvs of a derivation from the Nix language itself, we could just pass through that information as meta or similar. However, the only way to achieve this is via IFD and exportReferencesGraph, which is really slow. There are multiple proposals to allow eval-time querying of derivation inputs 1 2 3, but none of them went anywhere in the end.
• External tools which allow to query for input derivations (such as nix-eval-jobs) require writing drv files to disk to function, which would slow down CI by more than a factor of 2. Using tmpfs does help to an extent, but the slowdown is still over 80% from our measurements, which is unacceptable for CI running on every PR.

[infinisil]

First step towards implementing this now in NixOS/nixpkgs#531180!

Changed the approach to detect leaf-ness based on packages being referenced syntactically throughout Nixpkgs, that turns out to work decently well.