diff --git a/docs/concepts/priority.md b/docs/concepts/priority.md new file mode 100644 index 00000000..8e978cdf --- /dev/null +++ b/docs/concepts/priority.md @@ -0,0 +1,230 @@ +--- +title: Package Priority +description: How priority controls file conflict resolution in Flox environments +--- + +When multiple packages install a file to the same path, +Flox needs a rule to decide which one wins. +That rule is **priority**. + +## How priority works + +A Flox environment is a directory of symlinks +pointing into the Nix store. +When the environment is built, +every installed package's files are merged into this directory. +If two packages both provide `bin/python` or `share/licenses/LICENSE`, +their files collide. + +Priority resolves these collisions: + +- Every package has a `priority` value (default: **5**) +- **Lower numbers win**. + A package with `priority = 1` takes precedence + over a package with `priority = 5` +- If two packages collide at the **same priority**, + Flox reports an error and the environment fails to build + +Priority only affects files that actually collide. +Packages that install to non-overlapping paths coexist regardless +of their priority values. + +## Setting priority + +Set priority in the `[install]` section of your manifest: + +```toml +[install] +gcc.pkg-path = "gcc" + +gcc-unwrapped.pkg-path = "gcc-unwrapped" +gcc-unwrapped.priority = 6 +``` + +Here `gcc` keeps the default priority of 5 +and takes precedence over `gcc-unwrapped` (priority 6) +for any overlapping files. +Both packages are fully installed — +only the conflicting files are resolved in favor of `gcc`. + +## When you need to set priority + +Most packages don't conflict, +so you rarely need to think about priority. +The situations where it matters are: + +### Overlapping packages + +Some packages provide subsets or supersets of each other. +For example, +`gcc` and `gcc-unwrapped` both install some of the same files. +You may need both — +`gcc` for the compiler and `gcc-unwrapped.lib` for `libstdc++` — +but their overlapping files need a tiebreaker: + +```toml +[install] +gcc.pkg-path = "gcc" + +gcc-unwrapped.pkg-path = "gcc-unwrapped" +gcc-unwrapped.priority = 6 +gcc-unwrapped.pkg-group = "libraries" +``` + +### CUDA packages + +CUDA packages are a common source of collisions +because multiple packages install `LICENSE` files +to the same path. +When installing several CUDA packages, +assign incremental priorities: + +```toml +[install] +cuda_nvcc.pkg-path = "flox-cuda/cudaPackages_12_8.cuda_nvcc" +cuda_nvcc.systems = ["aarch64-linux", "x86_64-linux"] +cuda_nvcc.priority = 1 + +cuda_cudart.pkg-path = "flox-cuda/cudaPackages.cuda_cudart" +cuda_cudart.systems = ["aarch64-linux", "x86_64-linux"] +cuda_cudart.priority = 2 + +cudatoolkit.pkg-path = "flox-cuda/cudaPackages_12_8.cudatoolkit" +cudatoolkit.systems = ["aarch64-linux", "x86_64-linux"] +cudatoolkit.priority = 3 +``` + +The specific priority values don't matter +as long as each package gets a distinct number. +This tells Flox which `LICENSE` file to keep +when they collide. + +### Cross-platform fallbacks + +When providing platform-specific alternatives +(e.g. CUDA on Linux, CPU on macOS), +priority can indicate which variant is preferred +if both happen to be available: + +```toml +[install] +cuda-torch.pkg-path = "flox-cuda/python3Packages.torch" +cuda-torch.systems = ["x86_64-linux", "aarch64-linux"] +cuda-torch.priority = 1 + +torch-cpu.pkg-path = "python311Packages.torch-bin" +torch-cpu.systems = ["x86_64-darwin", "aarch64-darwin"] +torch-cpu.priority = 6 +``` + +In practice the `systems` filter already prevents collisions here, +but setting priority makes the intent explicit +and protects against future changes. + +## Diagnosing collisions + +When two packages collide at the same priority, +you'll see an error like: + +```text + > ❌ ERROR: 'cuda13.0-cuda_nvcc' conflicts with 'cuda13.0-libcublas'. Both packages provide the file 'LICENSE' + > Resolve by uninstalling one of the conflicting packages or setting the priority of the preferred package to a value lower than '5' +``` + +To fix this, +identify which package should win for the conflicting path +and give it a lower priority number: + +```toml +[install] +packageA.pkg-path = "packageA" +packageA.priority = 4 + +packageB.pkg-path = "packageB" +packageB.priority = 6 +``` + +## Priority and package groups + +Priority and [package groups][package-groups-concept] +solve different problems: + +- **Package groups** control which `nixpkgs` revision + each set of packages resolves against. + They address version and ABI compatibility. +- **Priority** controls which package's file wins + when two packages install to the same path. + It addresses file-level collisions + in the merged environment. + +You can use both together. +For example, +`gcc-unwrapped` might need its own package group +(to resolve against a different `nixpkgs` revision) +_and_ a higher priority number +(to let `gcc` win file conflicts): + +```toml +[install] +gcc.pkg-path = "gcc" + +gcc-unwrapped.pkg-path = "gcc-unwrapped" +gcc-unwrapped.priority = 6 +gcc-unwrapped.pkg-group = "libraries" +``` + +## Priority and builds + +Priority has no effect on what is available +during [manifest builds][manifest-builds-concept]. +Only packages in the `toplevel` +[package group][package-groups-concept] +are available during builds, +regardless of their priority. + +If you use `runtime-packages` to trim your build's closure, +all listed packages are included at their assigned priorities. + +## Priority and composition + +When [composing environments][composition-concept], +the included manifests are merged before the environment is built. +If two environments define the same install ID +(e.g. both provide `gcc.pkg-path`), +the later environment's package descriptor overrides the earlier one +during the manifest merge — +this is a manifest-level override, +not a file-level priority collision. + +After merging, +the resulting environment is built +and any file-level collisions between _different_ packages +are resolved by priority +the same way they are in a non-composed environment. + +## Reference + +| Aspect | Detail | +| ------ | ------ | +| Default value | `5` | +| Direction | Lower number = higher precedence | +| Valid values | Integers (no fixed bounds) | +| Common range | `1` through `10` | +| Same-priority collision | Error — environment fails to build | +| Scope | File paths only — doesn't affect resolution | + +### Nix equivalent + +Flox's `priority` maps directly to `meta.priority` in nixpkgs. +The semantics are identical: +default value of 5, lower wins, +same-priority collisions produce errors. + +Nixpkgs provides convenience wrappers — +`lib.meta.hiPrio` (sets priority to -10) and +`lib.meta.lowPrio` (sets priority to 10) — +but in Flox you set the integer directly in the manifest. + +[package-groups-concept]: ./package-groups.md +[manifest-builds-concept]: ./manifest-builds.md +[composition-concept]: ./composition.md diff --git a/docs/tutorials/vendoring-dependencies.md b/docs/tutorials/vendoring-dependencies.md new file mode 100644 index 00000000..360bd34c --- /dev/null +++ b/docs/tutorials/vendoring-dependencies.md @@ -0,0 +1,398 @@ +--- +title: Vendoring unavailable dependencies +description: How to build packages when dependencies aren't available in the Flox Catalog or nixpkgs +--- + +# Vendoring unavailable dependencies + +When [building with Flox][build-concept], +your dependencies come from different sources depending on which build type +you use: + +- **[Manifest builds][manifest-builds-concept]** draw installed packages from + the [Flox Catalog][catalog-concept] +- **[Nix expression builds][nix-expression-builds-concept]** draw packages from + [nixpkgs][base-catalog-concept] + +There are two distinct vendoring problems you may encounter; +each calls for a different solution. + +## Sandbox vendoring vs. missing dependencies + +**Sandbox vendoring** is when your dependencies exist in their language +ecosystem (Go modules, crates.io, npm) but a +[pure build][pure-builds-section] blocks network access, +preventing the build from downloading them. +The fix is a multi-stage build: +an impure first stage pre-fetches the dependencies, +and the pure second stage consumes them offline. +This is covered in the language guides +([Go][go-vendoring], [Rust][rust-vendoring], [Node.js][nodejs-vendoring]). + +**Missing dependency vendoring** is when the dependency doesn't exist +in the Flox Catalog or nixpkgs at all. +A toolchain may be too new for nixpkgs to have packaged, +a tool may not be packaged for Nix yet, +or a pre-built binary may be the only practical distribution method. +No amount of network access helps here +because the package manager itself doesn't know about the dependency: +you need to bring it into the ecosystem yourself. + +**This guide covers the second problem.** + +## Manifest builds + +Manifest builds install dependencies from the Flox Catalog +via the `[install]` section. +If a dependency isn't in the catalog, +you have several options. + +### Provide the dependency as a Nix expression build + +You can define missing dependencies as +[Nix expression builds][nix-expression-builds-concept] +in `.flox/pkgs/` +and install them into your environment alongside catalog packages. +Nix expression builds and manifest builds can coexist in the same environment, +so long as their package names don't conflict. + +For example, +if your project needs a dependency that isn't in the Flox Catalog, +you can package a pre-built binary as a Nix expression: + +```{ .nix .copy title=".flox/pkgs/my-custom-tool.nix" } +{ lib, stdenv, fetchurl, autoPatchelfHook }: + +let + sources = { + x86_64-linux = { + url = "https://example.com/releases/v1.0.0/my-custom-tool-linux-x64.tar.gz"; + hash = ""; + }; + aarch64-darwin = { + url = "https://example.com/releases/v1.0.0/my-custom-tool-darwin-arm64.tar.gz"; + hash = ""; + }; + }; + + platform = stdenv.hostPlatform.system; + source = sources.${platform} + or (throw "Unsupported platform: ${platform}"); +in +stdenv.mkDerivation { + pname = "my-custom-tool"; + version = "1.0.0"; + + src = fetchurl { + inherit (source) url hash; + }; + + nativeBuildInputs = lib.optionals stdenv.hostPlatform.isLinux [ + autoPatchelfHook + ]; + + dontConfigure = true; + dontBuild = true; + + installPhase = '' + mkdir -p $out/bin + tar -xzf $src + install -m755 my-custom-tool $out/bin/ + ''; +} +``` + +After running `git add .flox/pkgs/my-custom-tool.nix`, +the dependeny is available via `flox build my-custom-tool` +and can be installed into your environment. + +Use the [empty hash technique][nix-expression-hashes] +to determine the correct hash for each platform. + +### Use a flake input + +If the dependency is available as a Nix flake, +you can reference it directly in your manifest: + +```toml +[install] +my-tool.flake = "github:owner/repo" +``` + +This installs the flake's default package into your environment, +making it available to your manifest builds. + +### Download in an impure build stage + +For manifest builds with `sandbox = "off"` (the default), +you can download dependencies directly in your build script: + +```toml +[build.myproject] +command = ''' + # Download a tool not in the catalog + curl -L -o custom-tool \ + https://example.com/releases/custom-tool-linux-x86_64 + chmod +x custom-tool + export PATH="$PWD:$PATH" + + # Use it in the build + custom-tool generate ./src + mkdir -p $out/bin + cp result $out/bin/myproject +''' +``` + +!!! warning "Warning" + This approach only works with `sandbox = "off"` (the default). + [Pure builds][pure-builds-section] on Linux do not have network access. + If you need this dependency in a pure build, + use a multi-stage pattern where the first stage downloads the tool + impurely and the second stage builds with `sandbox = "pure"`: + + ```toml + [build.myproject-tools] + command = ''' + mkdir -p $out/bin + curl -L -o $out/bin/custom-tool \ + https://example.com/releases/custom-tool-linux-x86_64 + chmod +x $out/bin/custom-tool + ''' + + [build.myproject] + command = ''' + export PATH="${myproject-tools}/bin:$PATH" + custom-tool generate ./src + mkdir -p $out/bin + cp result $out/bin/myproject + ''' + sandbox = "pure" + ``` + +## Nix expression builds + +Nix expression builds draw all dependencies from nixpkgs +via function arguments. +When a dependency isn't available in nixpkgs +— or the version you need is too new — +you need to vendor it yourself. + +### Override an existing package's version + +If a package exists in nixpkgs but you need a newer version, +you can often override its `version` and `src` attributes: + +```{ .nix .copy title=".flox/pkgs/my-tool.nix" } +{ my-tool, fetchurl }: + +my-tool.overrideAttrs (finalAttrs: _oldAttrs: { + version = "2.5.0"; + src = fetchurl { + url = "https://example.com/releases/my-tool-${finalAttrs.version}.tar.gz"; + hash = ""; + }; +}) +``` + +This works well when the package's build process hasn't changed significantly +between versions. +See the [newer version example][nix-override-example] +in the Nix expression builds concept page for more details. + +### Vendor a pre-built toolchain + +When a toolchain version isn't in nixpkgs at all, +you can create a derivation that downloads pre-built binaries. +This is a common pattern for vendoring newer versions of Go, Node.js, +or other language runtimes. + +Here is an example that vendors a specific Go version +with cross-platform support: + +```{ .nix .copy title=".flox/pkgs/go_custom.nix" } +{ lib, stdenv, fetchurl }: + +let + version = "1.26.0"; + + sources = { + x86_64-linux = { + url = "https://go.dev/dl/go${version}.linux-amd64.tar.gz"; + hash = ""; + }; + aarch64-linux = { + url = "https://go.dev/dl/go${version}.linux-arm64.tar.gz"; + hash = ""; + }; + x86_64-darwin = { + url = "https://go.dev/dl/go${version}.darwin-amd64.tar.gz"; + hash = ""; + }; + aarch64-darwin = { + url = "https://go.dev/dl/go${version}.darwin-arm64.tar.gz"; + hash = ""; + }; + }; + + platform = stdenv.hostPlatform.system; + source = sources.${platform} + or (throw "Unsupported platform: ${platform}"); +in +stdenv.mkDerivation { + pname = "go"; + inherit version; + + src = fetchurl { + inherit (source) url hash; + }; + + dontConfigure = true; + dontBuild = true; + + installPhase = '' + mkdir -p $out + tar -xzf $src -C $out --strip-components=1 + ''; +} +``` + +Use the [empty hash technique][nix-expression-hashes] +to determine the correct hash for each platform. + +### Override a builder function with a custom toolchain + +Once you have a vendored toolchain, +you can override nixpkgs builder functions to use it +instead of the default. +For example, +to build a Go project that requires a newer Go than nixpkgs provides: + +```{ .nix .copy title=".flox/pkgs/my-go-project.nix" } +{ callPackage, buildGoModule, fetchFromGitHub }: + +let + customGo = callPackage ./go_custom.nix {}; + + buildGoModuleCustom = buildGoModule.override { + go = customGo; + }; +in +buildGoModuleCustom rec { + pname = "my-go-project"; + version = "1.0.0"; + + src = fetchFromGitHub { + owner = "example"; + repo = "my-go-project"; + rev = "v${version}"; + hash = ""; + }; + + vendorHash = ""; +} +``` + +This pattern works for any language ecosystem where nixpkgs provides +an overridable builder function: + +| Builder | Override pattern | +| ------- | --------------- | +| `buildGoModule` | `buildGoModule.override { go = customGo; }` | +| `buildNpmPackage` | `buildNpmPackage.override { nodejs = customNodejs; }` | +| `rustPlatform.buildRustPackage` | Build a custom `makeRustPlatform { rustc = ...; cargo = ...; }` | + +### Package a pre-built binary + +When building from source isn't practical +— for example, +when cargo vendoring fails due to complex git dependencies — +you can package a pre-built binary directly. + +For Linux binaries, +use `autoPatchelfHook` to automatically fix dynamic library references +so they resolve from the Nix store instead of system paths: + +```{ .nix .copy title=".flox/pkgs/my-tool.nix" } +{ lib, stdenv, fetchurl, autoPatchelfHook, gcc-unwrapped }: + +let + sources = { + x86_64-linux = { + url = "https://github.com/example/tool/releases/download/v1.0.0/tool-linux-x64.tar.gz"; + hash = ""; + }; + aarch64-linux = { + url = "https://github.com/example/tool/releases/download/v1.0.0/tool-linux-arm64.tar.gz"; + hash = ""; + }; + aarch64-darwin = { + url = "https://github.com/example/tool/releases/download/v1.0.0/tool-darwin-arm64.tar.gz"; + hash = ""; + }; + }; + + platform = stdenv.hostPlatform.system; + source = sources.${platform} + or (throw "Unsupported platform: ${platform}"); +in +stdenv.mkDerivation rec { + pname = "my-tool"; + version = "1.0.0"; + + src = fetchurl { + inherit (source) url hash; + }; + + nativeBuildInputs = lib.optionals stdenv.hostPlatform.isLinux [ + autoPatchelfHook + ]; + + buildInputs = lib.optionals stdenv.hostPlatform.isLinux [ + gcc-unwrapped.lib + ]; + + dontConfigure = true; + dontBuild = true; + + installPhase = '' + mkdir -p $out/bin + tar -xzf $src + install -m755 my-tool $out/bin/ + ''; +} +``` + +!!! note "Note" + `autoPatchelfHook` is Linux-only. + macOS binaries generally don't need patching, + but may require `__noChroot = true` if they depend on system frameworks + that the Nix sandbox blocks access to. + +!!! note "Note" + `gcc-unwrapped.lib` provides `libstdc++.so`, + which is needed by most C++ compiled binaries. + If the binary links against other shared libraries, + add those packages to `buildInputs` as well. + +## Choosing a strategy + +| Scenario | Build type | Recommended approach | +| -------- | ---------- | -------------------- | +| Language deps need pre-fetching for pure builds | Manifest | [Multi-stage vendoring][pure-builds-section] | +| Tool not in Flox Catalog | Manifest | [Nix expression build](#provide-the-dependency-as-a-nix-expression-build) or [flake input](#use-a-flake-input) | +| Newer version of existing nixpkgs package | Nix expression | [Override version and src](#override-an-existing-packages-version) | +| Toolchain version not in nixpkgs | Nix expression | [Vendor pre-built toolchain](#vendor-a-pre-built-toolchain) and [override builder](#override-a-builder-function-with-a-custom-toolchain) | +| Source build impractical | Nix expression | [Package pre-built binary](#package-a-pre-built-binary) | +| Dependency available as a Nix flake | Either | [Flake input](#use-a-flake-input) | + +[manifest-builds-concept]: ../concepts/manifest-builds.md +[nix-expression-builds-concept]: ../concepts/nix-expression-builds.md +[catalog-concept]: ../concepts/packages-and-catalog.md +[base-catalog-concept]: ../concepts/base-catalog.md +[pure-builds-section]: ../concepts/manifest-builds.md#pure-builds +[nix-expression-hashes]: ../concepts/nix-expression-builds.md#generating-hashes +[nix-override-example]: ../concepts/nix-expression-builds.md#example-newer-version-of-an-existing-package +[build-concept]: ../concepts/builds.md +[go-vendoring]: ../languages/go.md#vendoring-dependencies-in-pure-builds +[rust-vendoring]: ../languages/rust.md#vendoring-dependencies-in-pure-builds +[nodejs-vendoring]: ../languages/nodejs.md#vendoring-dependencies-in-pure-builds diff --git a/mkdocs.yml b/mkdocs.yml index d3111503..4da9a939 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -52,6 +52,7 @@ nav: - Designing cross-platform environments: tutorials/multi-arch-environments.md - Reusing and combining developer environments: tutorials/composition.md - Selecting package outputs: tutorials/package-outputs.md + - Vendoring unavailable dependencies: tutorials/vendoring-dependencies.md - Flox + CUDA: tutorials/cuda.md - Flox and systemd: tutorials/flox-and-systemd.md - Migration guides: