From 0b396892a6fdd7d16f77f2aa205ba4187d991123 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Sun, 19 Nov 2023 04:23:36 +0100 Subject: [PATCH 01/50] WIP: source file selection tutorial --- source/tutorials/index.md | 1 + source/tutorials/source-file-selection.md | 73 +++++++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 source/tutorials/source-file-selection.md diff --git a/source/tutorials/index.md b/source/tutorials/index.md index 90ed2749d2..5a5d2b274c 100644 --- a/source/tutorials/index.md +++ b/source/tutorials/index.md @@ -10,6 +10,7 @@ These sections contains series of lessons to get started. first-steps/index.md nix-language.md Packaging existing software +source-file-selection.md nixos/index.md cross-compilation.md ``` diff --git a/source/tutorials/source-file-selection.md b/source/tutorials/source-file-selection.md new file mode 100644 index 0000000000..042dff2145 --- /dev/null +++ b/source/tutorials/source-file-selection.md @@ -0,0 +1,73 @@ +(source-file-selection)= +# Source file selection + + +To build a local project in a Nix derivation, its source files must be accessible to the builder. +But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled), +it won't have access to the local project files by default. + +To make this work, the Nix language has certain builtin features to copy local paths to the Nix store, +whose paths are then accessible to derivation builders [^1]. + +[^1]: Technically only Nix store paths from the derivations inputs can be accessed, +but in practice this distinction is not important. + +These builtin features are very limited in functionality and are not recommended if you need to do anything more advanced. + +For advanced uses, use the file set library instead. + +## Builtins + +### Path coercion to strings + +The most transparent builtin feature is coercion of paths to strings, such as: +- Interpolating paths in strings: + ```nix + { runCommandCC }: + runCommandCC "test" { } '' + cc ${./main.c} -o $out + '' + ``` +- Using paths as derivation attributes: + ```nix + { stdenv }: + stdenv.mkDerivation { + name = "test"; + src = ./.; + } + ``` + +In both of these cases, the path gets imported into the Nix store +and transparently converted to its Nix store path. + +Some problems with this approach include: +- The base name of the path influences the store path, even for `./.`. +- All files in directories are unconditionally imported, even if they're unnecessary, impure or even meant to be private. + +### `builtins.path` + +The above problems can be fixed by using [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) instead. +It allows customising the name of the resulting store path with its `name` argument. +And it allows selecting the files that should be included with its `filter` argument. + +```nix +builtins.path { + name = "source"; + path = ./.; + filter = pathString: type: + baseNameOf pathString != "default.nix"; +``` + +This function is notoriously hard to use correctly by itself. + + From 197782d1b4210e1669f4aed54fc8f509f0084b61 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Sun, 19 Nov 2023 23:39:52 +0100 Subject: [PATCH 02/50] Updates --- source/tutorials/source-file-selection.md | 214 +++++++++++++++++++--- 1 file changed, 191 insertions(+), 23 deletions(-) diff --git a/source/tutorials/source-file-selection.md b/source/tutorials/source-file-selection.md index 042dff2145..6a28811082 100644 --- a/source/tutorials/source-file-selection.md +++ b/source/tutorials/source-file-selection.md @@ -6,45 +6,199 @@ To build a local project in a Nix derivation, its source files must be accessibl But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled), it won't have access to the local project files by default. -To make this work, the Nix language has certain builtin features to copy local paths to the Nix store, +To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store, whose paths are then accessible to derivation builders [^1]. [^1]: Technically only Nix store paths from the derivations inputs can be accessed, but in practice this distinction is not important. -These builtin features are very limited in functionality and are not recommended if you need to do anything more advanced. +These builtin features are very limited in functionality and are not recommended for anything non-trivial. For more advanced use cases, the file set library should be used instead. -For advanced uses, use the file set library instead. +In this tutorial you'll learn both how to use the builtins and the file set library. -## Builtins +## Setting up a local experiment -### Path coercion to strings +To experiment with source file selection, we'll set up a local project. -The most transparent builtin feature is coercion of paths to strings, such as: -- Interpolating paths in strings: +To start out, create a new directory, enter it, and set up `niv` to manage the Nixpkgs dependency: +```shell-session +$ mkdir select +$ cd select +$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable" +``` + + +:::{note} +For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. +::: + +Then create a `default.nix` file with these contents: +```nix +{ + system ? builtins.currentSystem, + sources ? import ./nix/sources.nix, +}: +let + pkgs = import sources.nixpkgs { + # Ensure purity + config = { }; + overlays = [ ]; + inherit system; + }; +in +pkgs.callPackage ./package.nix { } +``` + +In this tutorial we'll experiment with different `package.nix` contents, while keeping `default.nix` the same. + +For now, let's have a simple `package.nix` to verify everything works so far: + +```nix +{ runCommand }: +runCommand "hello" { } '' + echo hello world +'' +``` + +And try it out: +```shell-session +$ nix-build +this derivation will be built: + /nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv +building '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'... +hello world +error: builder for '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv' + failed to produce output path for output 'out' +``` + +We could also add `touch $out` to make the build succeed, +but we'll omit that for the sake of the tutorial, since we only need the build logs. +This also makes it easier to build it again, since successful derivation builds would get cached. +From now on we'll also make build outputs a bit shorter for the sake of brevity. + +## Builtin coercion of paths to strings + +The easiest way to use local files in builds is using the built-in coercion of [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) to strings. + +Let's create a local `string.txt` file: +``` +$ echo "This is a string" > string.txt +``` + +:::{note} +Flakes in Git directories requires git add. +::: + +The two main ways to coerce paths to strings are: +- Interpolating paths in strings. To try that, change your `package.nix` file to: + ```nix + { runCommand }: + runCommand "file-interpolation" { } '' + ( + set -x + cat ${./string.txt} + ) + '' + ``` + + :::{note} + Interpolation into bash scripts generally requires [`lib.escapeShellArg`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.strings.escapeShellArg) for correct escaping. + In this case however, the interpolation results in a Nix store path of the form `/nix/store/-`, + and all valid characters of such store paths don't need to be escaped in bash. + ::: + +- Using paths as derivation attributes. To try that, change your `package.nix` file to: ```nix - { runCommandCC }: - runCommandCC "test" { } '' - cc ${./main.c} -o $out + { runCommand }: + runCommand "file-interpolation" { + stringFile = ./string.txt; + } '' + ( + set -x + cat "$stringFile" + ) '' ``` -- Using paths as derivation attributes: + + :::{note} + Nowadays using the explicit `env` attribute is recommended to set environment variables. + `env` doesn't implicitly coerce paths to strings, so it requires using string intepolation instead: ```nix - { stdenv }: - stdenv.mkDerivation { - name = "test"; - src = ./.; - } + { runCommand }: + runCommand "file-interpolation" { + env.stringFile = "${./string.txt}"; + } '' + ( + set -x + cat "$stringFile" + ) + '' ``` + ::: + +These all do the same when built: +```shell-session +$ nix-build +building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-interpolation.drv'... +++ cat /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt +This is a string +``` + +As you can see, the `string.txt` file was hashed and added to the store, +which then allowed the build to access it. -In both of these cases, the path gets imported into the Nix store -and transparently converted to its Nix store path. +The underlying functionality is the same as `nix-store --add` on an absolute path: +```shell-session +$ nix-store --add $(pwd)/string.txt +/nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt +``` -Some problems with this approach include: -- The base name of the path influences the store path, even for `./.`. -- All files in directories are unconditionally imported, even if they're unnecessary, impure or even meant to be private. +## Built-in path coercion on directories -### `builtins.path` +This path coercion also works on directories the same as it does on files, let's try it out: + +```nix +{ runCommand, tree }: +runCommand "directory-interpolation" { + # To nicely show path contents + nativeBuildInputs = [ tree ]; +} '' + tree ${./.} +'' +``` + +Running it gives us: +```shell-session +$ nix-build +building '/nix/store/6ybg4v48xy8azhrnfdccdmhd2gr938f5-directory-interpolation.drv'... +/nix/store/xdfchqpfx20ar9jil9kys99wc6hnm9zx-select +|-- default.nix +|-- nix +| |-- sources.json +| `-- sources.nix +|-- package.nix +`-- string.txt +``` + +But here we can get into some subtle trouble: +- Note how the name of the store path ends with `-select`. + So the name of the local directory influenced the result. + + This means that whenever you rename the project directory + or a collegue runs it in a different directory name, + you're going to get different build results! + +- All files in the directory are unconditionally added to the Nix store. + + This means that: + - Even if your build only needs a few files, + changing _any_ file in the directory requires rebuilding the derivation, + potentially wasting a lot of time. + + - If you have any secrets stored in the current directory, + they get imported into the Nix store too, exposing them to all users on the system! + +## `builtins.path` The above problems can be fixed by using [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) instead. It allows customising the name of the resulting store path with its `name` argument. @@ -58,7 +212,7 @@ builtins.path { baseNameOf pathString != "default.nix"; ``` -This function is notoriously hard to use correctly by itself. +However, this function is notoriously hard to use correctly by itself. From 836b6cde50cf5e229afd67fcfd442c33805f8222 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Mon, 20 Nov 2023 00:38:18 +0100 Subject: [PATCH 03/50] Updates --- source/tutorials/source-file-selection.md | 92 ++++++++++++++++++----- 1 file changed, 75 insertions(+), 17 deletions(-) diff --git a/source/tutorials/source-file-selection.md b/source/tutorials/source-file-selection.md index 6a28811082..c82a1480fb 100644 --- a/source/tutorials/source-file-selection.md +++ b/source/tutorials/source-file-selection.md @@ -85,15 +85,11 @@ Let's create a local `string.txt` file: $ echo "This is a string" > string.txt ``` -:::{note} -Flakes in Git directories requires git add. -::: - The two main ways to coerce paths to strings are: - Interpolating paths in strings. To try that, change your `package.nix` file to: ```nix { runCommand }: - runCommand "file-interpolation" { } '' + runCommand "file-coercion" { } '' ( set -x cat ${./string.txt} @@ -110,7 +106,7 @@ The two main ways to coerce paths to strings are: - Using paths as derivation attributes. To try that, change your `package.nix` file to: ```nix { runCommand }: - runCommand "file-interpolation" { + runCommand "file-coercion" { stringFile = ./string.txt; } '' ( @@ -125,7 +121,7 @@ The two main ways to coerce paths to strings are: `env` doesn't implicitly coerce paths to strings, so it requires using string intepolation instead: ```nix { runCommand }: - runCommand "file-interpolation" { + runCommand "file-coercion" { env.stringFile = "${./string.txt}"; } '' ( @@ -139,7 +135,7 @@ The two main ways to coerce paths to strings are: These all do the same when built: ```shell-session $ nix-build -building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-interpolation.drv'... +building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-coercion.drv'... ++ cat /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt This is a string ``` @@ -159,7 +155,7 @@ This path coercion also works on directories the same as it does on files, let's ```nix { runCommand, tree }: -runCommand "directory-interpolation" { +runCommand "directory-coercion" { # To nicely show path contents nativeBuildInputs = [ tree ]; } '' @@ -180,7 +176,41 @@ building '/nix/store/6ybg4v48xy8azhrnfdccdmhd2gr938f5-directory-interpolation.dr `-- string.txt ``` -But here we can get into some subtle trouble: +It's also very common to use this for the [`src`](https://nixos.org/manual/nixpkgs/stable/#var-stdenv-src) variable in `stdenv.mkDerivation` like so: + +```nix +{ stdenv, tree }: +stdenv.mkDerivation { + name = "directory-coercion"; + src = ./.; + nativeBuildInputs = [ tree ]; + postInstall = '' + touch created-in-the-build + tree + ''; +} +``` + +Setting `src` will copy the resulting store path into the build directory and mark it as mutable. +For the many commands that expect to be able to write to the current directory, this is great: + +```shell-session +$ nix-build +building '/nix/store/2cqd93fpnb4vqwkwmbl66dbxhndq1vhh-directory-coercion.drv'... +unpacking sources +unpacking source archive /nix/store/178fbwa8iwdl6b85yafksdbwlxf6mjca-select +[...] +. +|-- created-in-the-build +|-- default.nix +|-- nix +| |-- sources.json +| `-- sources.nix +|-- package.nix +`-- string.txt +``` + +However there are some subtle problems with this approach: - Note how the name of the store path ends with `-select`. So the name of the local directory influenced the result. @@ -195,24 +225,52 @@ But here we can get into some subtle trouble: changing _any_ file in the directory requires rebuilding the derivation, potentially wasting a lot of time. - - If you have any secrets stored in the current directory, + - If you have any secrets in the current directory, they get imported into the Nix store too, exposing them to all users on the system! ## `builtins.path` + + The above problems can be fixed by using [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) instead. It allows customising the name of the resulting store path with its `name` argument. And it allows selecting the files that should be included with its `filter` argument. ```nix -builtins.path { - name = "source"; - path = ./.; - filter = pathString: type: - baseNameOf pathString != "default.nix"; +{ runCommand, tree, lib }: +let + source = builtins.path { + # The convention is to use "source" + name = "source"; + path = ./.; + + filter = pathString: type: + # Recurse into directories + type == "directory" + # Don't include .nix files + || ! lib.hasSuffix ".nix" pathString; + }; +in +runCommand "builtins-path" { + nativeBuildInputs = [ tree ]; +} '' + tree ${source} +'' +``` + +The shown filter will recurse into all directories and filter out all `.nix` files: + +```shell-session +$ nix-build +building '/nix/store/3x051rr6fainqi3a4mmmb06145m0j0mw-builtins-path.drv'... +/nix/store/95mlqjmm13vd4ambw2pac5gj6i4wxcx4-source +|-- nix +| `-- sources.json +`-- string.txt ``` -However, this function is notoriously hard to use correctly by itself. +Writing more complex `filter` functions however is notoriously tricky, +which is why it's not recommended to use this function directly. + + To build a local project in a Nix derivation, its source files must be accessible to the builder. But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled), it won't have access to the local project files by default. @@ -27,7 +29,6 @@ $ cd select $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable" ``` - :::{note} For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. ::: @@ -46,7 +47,10 @@ let inherit system; }; in -pkgs.callPackage ./package.nix { } +{ + package = pkgs.callPackage ./package.nix { }; + inherit pkgs; +} ``` In this tutorial we'll experiment with different `package.nix` contents, while keeping `default.nix` the same. @@ -62,7 +66,7 @@ runCommand "hello" { } '' And try it out: ```shell-session -$ nix-build +$ nix-build -A package this derivation will be built: /nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv building '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'... @@ -76,7 +80,9 @@ but we'll omit that for the sake of the tutorial, since we only need the build l This also makes it easier to build it again, since successful derivation builds would get cached. From now on we'll also make build outputs a bit shorter for the sake of brevity. -## Builtin coercion of paths to strings +## Builtins + +### Coercion of paths to strings The easiest way to use local files in builds is using the built-in coercion of [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) to strings. @@ -134,7 +140,7 @@ The two main ways to coerce paths to strings are: These all do the same when built: ```shell-session -$ nix-build +$ nix-build -A package building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-coercion.drv'... ++ cat /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt This is a string @@ -149,7 +155,7 @@ $ nix-store --add $(pwd)/string.txt /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt ``` -## Built-in path coercion on directories +### Coercion of directory paths to strings This path coercion also works on directories the same as it does on files, let's try it out: @@ -165,7 +171,7 @@ runCommand "directory-coercion" { Running it gives us: ```shell-session -$ nix-build +$ nix-build -A package building '/nix/store/6ybg4v48xy8azhrnfdccdmhd2gr938f5-directory-interpolation.drv'... /nix/store/xdfchqpfx20ar9jil9kys99wc6hnm9zx-select |-- default.nix @@ -195,7 +201,7 @@ Setting `src` will copy the resulting store path into the build directory and ma For the many commands that expect to be able to write to the current directory, this is great: ```shell-session -$ nix-build +$ nix-build -A package building '/nix/store/2cqd93fpnb4vqwkwmbl66dbxhndq1vhh-directory-coercion.drv'... unpacking sources unpacking source archive /nix/store/178fbwa8iwdl6b85yafksdbwlxf6mjca-select @@ -228,7 +234,7 @@ However there are some subtle problems with this approach: - If you have any secrets in the current directory, they get imported into the Nix store too, exposing them to all users on the system! -## `builtins.path` +### `builtins.path` @@ -239,7 +245,7 @@ And it allows selecting the files that should be included with its `filter` argu ```nix { runCommand, tree, lib }: let - source = builtins.path { + src = builtins.path { # The convention is to use "source" name = "source"; path = ./.; @@ -251,17 +257,20 @@ let || ! lib.hasSuffix ".nix" pathString; }; in -runCommand "builtins-path" { +stdenv.mkDerivation { + name = "builtins-path"; + inherit src; nativeBuildInputs = [ tree ]; -} '' - tree ${source} -'' + postInstall = '' + tree + ''; +} ``` The shown filter will recurse into all directories and filter out all `.nix` files: ```shell-session -$ nix-build +$ nix-build -A package building '/nix/store/3x051rr6fainqi3a4mmmb06145m0j0mw-builtins-path.drv'... /nix/store/95mlqjmm13vd4ambw2pac5gj6i4wxcx4-source |-- nix @@ -272,6 +281,147 @@ building '/nix/store/3x051rr6fainqi3a4mmmb06145m0j0mw-builtins-path.drv'... Writing more complex `filter` functions however is notoriously tricky, which is why it's not recommended to use this function directly. +## File sets + +A better way to select local files is to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset). +This library is an abstraction built on top of `builtins.path`, with essentially the same functionality, but with an easier and safer interface. + +The library is based on the concept of file sets, +a data type representing a collection of local files. +File sets can be created, composed and used with the various functions of the library. + +The easiest way to experiment with the file set library is to first use it through `nix repl`. +We can use the `pkgs` attribute exposed by `default.nix` +to define `fs = pkgs.lib.fileset` as a shorthand alias for the library: + +```shell-session +$ nix repl -f . +[...] +nix-repl> fs = pkgs.lib.fileset +``` + +### Basics + +It's probably the easiest to just jump right in +by using the [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function, +which pretty-prints the files included in a given file set: + +```shell-session +nix-repl> fs.trace ./. null +trace: /home/user/select (all files in directory) +null +``` + +You might wonder where the file set here is, because we just passed a _path_ to the function! + +The key is that for all file set functions that expect a file set for an argument, it also accepts a path instead. +Such a path argument is then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) +to a file set that contains _all files under the given path_. +We can see this from the trace `/home/user/select (all files in directory)` + +Note that in contrast to [coercion of paths to strings](#coercion-of-paths-to-strings), +file sets _never_ add their files to the Nix store unless explicitly requested. +So you don't have to worry about accidentally copying secrets into the store. + +:::{tip} +The `trace` function pretty-prints its first agument and returns its second argument. +But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: + +```shell-session +nix-repl> fs.trace ./. +trace: /home/user/select (all files in directory) +«lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8» +``` +::: + +This implicit coercion also works for files: + +```shell-session +nix-repl> fs.trace ./string.txt +trace: /home/user/select +trace: - string.txt (regular) +``` + +We can see that in addition to the included file, +it also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). + +And if we make a typo for a path that doesn't exist, the file set library adequately complains about it: + +```shell-session +nix-repl> fs.trace ./string.nix +error: lib.fileset.trace: Argument (/home/user/select/string.nix) + is a path that does not exist. +``` + +### Adding files to the Nix store + +The file set library wouldn't be very useful if you couldn't also add its files to the Nix store for use in derivations. +That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in. +It allows us to create a Nix store path containing exactly the files in the file set, with its root at a specific path. + +Let's switch back to editing our `package.nix` file to try it out: + +```nix +{ stdenv, tree, lib }: +let + fs = lib.fileset; +in +stdenv.mkDerivation { + name = "filesets"; + src = fs.toSource { + root = ./.; + fileset = ./string.txt; + }; + nativeBuildInputs = [ tree ]; + postInstall = '' + tree + ''; +} +``` + +Building this, we can see that indeed only `string.nix` was included: +``` +$ nix-build +building '/nix/store/1cdgj8b1jrg2k751jidj9564r66lpvj9-filesets.drv'... +unpacking sources +unpacking source archive /nix/store/083k2phzhdakc649z5ql8f7cyisws6wp-source +[...] +. +`-- string.txt +``` + +We can also avoid using `tree` by using the `trace` function instead: + +```nix +{ stdenv, lib }: +let + fs = lib.fileset; + sourceFiles = ./string.txt; +in +fs.trace sourceFiles +stdenv.mkDerivation { + name = "filesets"; + src = fs.toSource { + root = ./.; + fileset = sourceFiles; + }; +} +``` + +And building it: + +``` +$ nix-build +trace: /home/user/select +trace: - string.txt (regular) +``` + +### Combinators + +The real benefit of the file set library lies in its combinator functions. + + + , +so let's try it: + +```nix + sourceFiles = fs.difference ./. (fs.maybeMissing ./result); +``` + +This now works, reliably filtering out `./result` if it exists: + +``` +$ nix-build -A package +trace: /home/user/select (all files in directory) +this derivation will be built: + /nix/store/ygpx17kshzc6bj3c71xlda8szw6qi1sr-filesets.drv +[...] +/nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets + +$ nix-build -A package +trace: /home/user/select +trace: - default.nix (regular) +trace: - nix (all files in directory) +trace: - package.nix (regular) +trace: - string.txt (regular) +/nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets +``` + +### Adding file sets together + +We still have a problem however: +Changing _any_ of the included files causes the derivation to be rebuilt, +even though it doesn't depend on those files. + +```shell-session +$ echo "# Just a comment" >> package.nix + +$ nix-build -A package +trace: /home/tweagysil/select +trace: - default.nix (regular) +trace: - nix (all files in directory) +trace: - package.nix (regular) +trace: - string.txt (regular) +this derivation will be built: + /nix/store/zmgpqlpfz2jq0w9rdacsnpx8ni4n77cn-filesets.drv +[...] +/nix/store/6pffjljjy3c7kla60nljk3fad4q4kkzn-filesets +``` + +One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions) to create a file set combined of all of the files we don't want: + +```nix + sourceFiles = + fs.difference + ./. + (fs.unions [ + (fs.maybeMissing ./result) + ./default.nix + ./package.nix + ./nix + ]); +``` + +This also gives us the opportunity to show the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function, +which as the name implies, allows filtering the files in a local path. +We use it to select all files whose name ends with `.nix`: + +```nix + sourceFiles = + fs.difference + ./. + (fs.unions [ + (fs.maybeMissing ./result) + (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) + ./nix + ]); +``` + +Changing any of the removed files now doesn't necessarily require a rebuild anymore: + +```shell-session +$ nix-build -A package +trace: /home/user/select +trace: - string.txt (regular) +/nix/store/clrd19vn5cv6n7x7hzajq1fv43qig7cp-filesets + +$ echo "# Just a comment" >> package.nix + +$ nix-build -A package +trace: /home/user/select +trace: - string.txt (regular) +/nix/store/clrd19vn5cv6n7x7hzajq1fv43qig7cp-filesets +``` + +Notable with this approach is that new files added to the current directory are **included by default**. +Depending on your project, this might be a better fit than the alternative in the next section. + +### Only including necessary files + +Let's create some extra files, and make the build use them all - +(file-sets)= +# File sets To build a local project in a Nix derivation, its source files must be accessible to the builder. @@ -10,298 +8,29 @@ it won't have access to the local project files by default. To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store, whose paths are then accessible to derivation builders [^1]. +However using the builtin features directly can be very tricky. [^1]: Technically only Nix store paths from the derivations inputs can be accessed, but in practice this distinction is not important. -These builtin features are very limited in functionality and are not recommended for anything non-trivial. For more advanced use cases, the file set library should be used instead. - -In this tutorial you'll learn both how to use the builtins and the file set library. - -## Setting up a local experiment - -To experiment with source file selection, we'll set up a local project. - -To start out, create a new directory, enter it, and set up `niv` to manage the Nixpkgs dependency: -```shell-session -$ mkdir select -$ cd select -$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable" -``` - -:::{note} -For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. -::: - -Then create a `default.nix` file with these contents: -```nix -{ - system ? builtins.currentSystem, - sources ? import ./nix/sources.nix, -}: -let - pkgs = import sources.nixpkgs { - # Ensure purity - config = { }; - overlays = [ ]; - inherit system; - }; -in -{ - package = pkgs.callPackage ./package.nix { }; - inherit pkgs; -} -``` - -In this tutorial we'll experiment with different `package.nix` contents, while keeping `default.nix` the same. - -For now, let's have a simple `package.nix` to verify everything works so far: - -```nix -{ runCommand }: -runCommand "hello" { } '' - echo hello world -'' -``` - -And try it out: -```shell-session -$ nix-build -A package -this derivation will be built: - /nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv -building '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'... -hello world -error: builder for '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv' - failed to produce output path for output 'out' -``` - -We could also add `touch $out` to make the build succeed, -but we'll omit that for the sake of the tutorial, since we only need the build logs. -This also makes it easier to build it again, since successful derivation builds would get cached. -From now on we'll also make build outputs a bit shorter for the sake of brevity. - -## Builtins - -### Coercion of paths to strings - -The easiest way to use local files in builds is using the built-in coercion of [paths](https://nixos.org/manual/nix/stable/language/values.html#type-path) to strings. - -Let's create a local `string.txt` file: -``` -$ echo "This is a string" > string.txt -``` - -The two main ways to coerce paths to strings are: -- Interpolating paths in strings. To try that, change your `package.nix` file to: - ```nix - { runCommand }: - runCommand "file-coercion" { } '' - ( - set -x - cat ${./string.txt} - ) - '' - ``` - - :::{note} - Interpolation into bash scripts generally requires [`lib.escapeShellArg`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.strings.escapeShellArg) for correct escaping. - In this case however, the interpolation results in a Nix store path of the form `/nix/store/-`, - and all valid characters of such store paths don't need to be escaped in bash. - ::: - -- Using paths as derivation attributes. To try that, change your `package.nix` file to: - ```nix - { runCommand }: - runCommand "file-coercion" { - stringFile = ./string.txt; - } '' - ( - set -x - cat "$stringFile" - ) - '' - ``` - - :::{note} - Nowadays using the explicit `env` attribute is recommended to set environment variables. - `env` doesn't implicitly coerce paths to strings, so it requires using string intepolation instead: - ```nix - { runCommand }: - runCommand "file-coercion" { - env.stringFile = "${./string.txt}"; - } '' - ( - set -x - cat "$stringFile" - ) - '' - ``` - ::: - -These all do the same when built: -```shell-session -$ nix-build -A package -building '/nix/store/9fi0khrkmqw5srjzjsfa0b05hf8div4c-file-coercion.drv'... -++ cat /nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt -This is a string -``` - -As you can see, the `string.txt` file was hashed and added to the store, -which then allowed the build to access it. - -The underlying functionality is the same as `nix-store --add` on an absolute path: -```shell-session -$ nix-store --add $(pwd)/string.txt -/nix/store/j5lwpnlfrngks3bpidfr5hcrhgq0fy78-string.txt -``` - -### Coercion of directory paths to strings - -This path coercion also works on directories the same as it does on files, let's try it out: - -```nix -{ runCommand, tree }: -runCommand "directory-coercion" { - # To nicely show path contents - nativeBuildInputs = [ tree ]; -} '' - tree ${./.} -'' -``` - -Running it gives us: -```shell-session -$ nix-build -A package -building '/nix/store/6ybg4v48xy8azhrnfdccdmhd2gr938f5-directory-interpolation.drv'... -/nix/store/xdfchqpfx20ar9jil9kys99wc6hnm9zx-select -|-- default.nix -|-- nix -| |-- sources.json -| `-- sources.nix -|-- package.nix -`-- string.txt -``` - -It's also very common to use this for the [`src`](https://nixos.org/manual/nixpkgs/stable/#var-stdenv-src) variable in `stdenv.mkDerivation` like so: - -```nix -{ stdenv, tree }: -stdenv.mkDerivation { - name = "directory-coercion"; - src = ./.; - nativeBuildInputs = [ tree ]; - postInstall = '' - touch created-in-the-build - tree - ''; -} -``` - -Setting `src` will copy the resulting store path into the build directory and mark it as mutable. -For the many commands that expect to be able to write to the current directory, this is great: +In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. +It abstracts over the builtin features with essentially the same functionality, +but an easier and safer interface. -```shell-session -$ nix-build -A package -building '/nix/store/2cqd93fpnb4vqwkwmbl66dbxhndq1vhh-directory-coercion.drv'... -unpacking sources -unpacking source archive /nix/store/178fbwa8iwdl6b85yafksdbwlxf6mjca-select -[...] -. -|-- created-in-the-build -|-- default.nix -|-- nix -| |-- sources.json -| `-- sources.nix -|-- package.nix -`-- string.txt -``` - -However there are some subtle problems with this approach: -- Note how the name of the store path ends with `-select`. - So the name of the local directory influenced the result. - - This means that whenever you rename the project directory - or a collegue runs it in a different directory name, - you're going to get different build results! - -- All files in the directory are unconditionally added to the Nix store. - - This means that: - - Even if your build only needs a few files, - changing _any_ file in the directory requires rebuilding the derivation, - potentially wasting a lot of time. - - - If you have any secrets in the current directory, - they get imported into the Nix store too, exposing them to all users on the system! - -### `builtins.path` - - - -The above problems can be fixed by using [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) instead. -It allows customising the name of the resulting store path with its `name` argument. -And it allows selecting the files that should be included with its `filter` argument. - -```nix -{ runCommand, tree, lib }: -let - src = builtins.path { - # The convention is to use "source" - name = "source"; - path = ./.; - - filter = pathString: type: - # Recurse into directories - type == "directory" - # Don't include .nix files - || ! lib.hasSuffix ".nix" pathString; - }; -in -stdenv.mkDerivation { - name = "builtins-path"; - inherit src; - nativeBuildInputs = [ tree ]; - postInstall = '' - tree - ''; -} -``` - -The shown filter will recurse into all directories and filter out all `.nix` files: +## Basics -```shell-session -$ nix-build -A package -building '/nix/store/3x051rr6fainqi3a4mmmb06145m0j0mw-builtins-path.drv'... -/nix/store/95mlqjmm13vd4ambw2pac5gj6i4wxcx4-source -|-- nix -| `-- sources.json -`-- string.txt -``` - -Writing more complex `filter` functions however is notoriously tricky, -which is why it's not recommended to use this function directly. - -## File sets - -A better way to select local files is to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset). -This library is an abstraction built on top of `builtins.path`, with essentially the same functionality, but with an easier and safer interface. - -The library is based on the concept of file sets, -a data type representing a collection of local files. +The file set library is based on the concept of _file sets_, +a conceptual data type representing a collection of local files. File sets can be created, composed and used with the various functions of the library. The easiest way to experiment with the file set library is to first use it through `nix repl`. -We can use the `pkgs` attribute exposed by `default.nix` -to define `fs = pkgs.lib.fileset` as a shorthand alias for the library: ```shell-session -$ nix repl -f . +$ nix repl -f channel:nixos-unstable [...] -nix-repl> fs = pkgs.lib.fileset +nix-repl> fs = lib.fileset ``` -### Basics - It's probably the easiest to just jump right in by using the [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function, which pretty-prints the files included in a given file set: @@ -314,13 +43,12 @@ null You might wonder where the file set here is, because we just passed a _path_ to the function! -The key is that for all file set functions that expect a file set for an argument, it also accepts a path instead. -Such a path argument is then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) -to a file set that contains _all files under the given path_. +The key is that for all functions that expect a file set for an argument, they also accepts paths instead. +Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) +to file sets containing _all files under the given path_. We can see this from the trace `/home/user/select (all files in directory)` -Note that in contrast to [coercion of paths to strings](#coercion-of-paths-to-strings), -file sets _never_ add their files to the Nix store unless explicitly requested. +File sets _never_ add their files to the Nix store unless explicitly requested. So you don't have to worry about accidentally copying secrets into the store. :::{tip} @@ -353,44 +81,73 @@ error: lib.fileset.trace: Argument (/home/user/select/string.nix) is a path that does not exist. ``` -### Adding files to the Nix store +## Setting up a local experiment -The file set library wouldn't be very useful if you couldn't also add its files to the Nix store for use in derivations. -That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in. -It allows us to create a Nix store path containing exactly the files in the file set, with its root at a specific path. +To further experiment with the library, let's set up a local project. -Let's switch back to editing our `package.nix` file to try it out: +To start out, create a new directory, enter it, and set up `niv` to pin the Nixpkgs dependency: +```shell-session +$ mkdir select +$ cd select +$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable" +``` +:::{note} +For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. +::: + +Then create a `default.nix` file with these contents: ```nix -{ stdenv, tree, lib }: +{ + system ? builtins.currentSystem, + sources ? import ./nix/sources.nix, +}: let - fs = lib.fileset; -in -stdenv.mkDerivation { - name = "filesets"; - src = fs.toSource { - root = ./.; - fileset = ./string.txt; + pkgs = import sources.nixpkgs { + # Ensure purity + config = { }; + overlays = [ ]; + inherit system; }; - nativeBuildInputs = [ tree ]; - postInstall = '' - tree - ''; -} +in +pkgs.callPackage ./package.nix { } ``` -Building this, we can see that indeed only `string.nix` was included: +From now on we'll just change the contents of `package.nix` while keeping `default.nix` the same. + +For now, let's have a simple `package.nix` to verify everything works so far: + +```nix +{ runCommand }: +runCommand "hello" { } '' + echo hello world +'' ``` + +And try it out: +```shell-session $ nix-build -building '/nix/store/1cdgj8b1jrg2k751jidj9564r66lpvj9-filesets.drv'... -unpacking sources -unpacking source archive /nix/store/083k2phzhdakc649z5ql8f7cyisws6wp-source -[...] -. -`-- string.txt +this derivation will be built: + /nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv +building '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv'... +hello world +error: builder for '/nix/store/kmf9sw8fn7ps3ndqs31hvqwsa35b8l3g-hello.drv' + failed to produce output path for output 'out' ``` -We can also avoid using `tree` by using the `trace` function instead: +We could also add `touch $out` to make the build succeed, +but we'll omit that for the sake of the tutorial, since we only need the build logs. +This also makes it easier to build it again, since successful derivation builds would get cached. +From now on we'll also make build outputs a bit shorter for the sake of brevity. + +## Adding files to the Nix store + +The file set library wouldn't be very useful if you couldn't also add its files to the Nix store for use in derivations. +That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in. +It allows us to create a Nix store path containing exactly only the files that are in the file set, +added to the Nix store rooted at a specific path. + +Let's try it out by defining `package.nix` as follows: ```nix { stdenv, lib }: @@ -416,10 +173,10 @@ trace: /home/user/select trace: - string.txt (regular) ``` -The real benefit of the file set library lies in its combinator functions. +But the real benefit of the file set library lies in its combinator functions. These allow you to compose file sets in different ways to achieve complex behavior. -### Avoiding unnecessary rebuilds +## Avoiding unnecessary rebuilds To show some more motivation, let's first use the file set library to include all files in the local directory in the build, @@ -538,7 +295,7 @@ trace: - string.txt (regular) /nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets ``` -### Adding file sets together +## Adding file sets together We still have a problem however: Changing _any_ of the included files causes the derivation to be rebuilt, @@ -559,7 +316,8 @@ this derivation will be built: /nix/store/6pffjljjy3c7kla60nljk3fad4q4kkzn-filesets ``` -One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions) to create a file set combined of all of the files we don't want: +One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions) +to create a file set containing all of the files we don't want: ```nix sourceFiles = @@ -607,25 +365,17 @@ trace: - string.txt (regular) Notable with this approach is that new files added to the current directory are **included by default**. Depending on your project, this might be a better fit than the alternative in the next section. -### Only including necessary files +## Git -Let's create some extra files, and make the build use them all +## Only including necessary files +For the complete list and more details, see the [reference documentation](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset). From 33ff09d6cd97818aa565be3a576eaf727157c69c Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Tue, 21 Nov 2023 03:35:35 +0100 Subject: [PATCH 09/50] Polish --- source/conf.py | 2 +- source/tutorials/file-sets.md | 105 ++++++++++++++++++++++------------ 2 files changed, 71 insertions(+), 36 deletions(-) diff --git a/source/conf.py b/source/conf.py index 76c0aaff40..b62826e4a3 100644 --- a/source/conf.py +++ b/source/conf.py @@ -59,7 +59,7 @@ myst_number_code_blocks = [ "nix", "python" ] -copybutton_prompt_text = r"\$ " +copybutton_prompt_text = r"\$ |nix-repl> " copybutton_prompt_is_regexp = True # Add any paths that contain templates here, relative to this directory. diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 6fa4756ac1..aa785c2627 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -8,7 +8,7 @@ it won't have access to the local project files by default. To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store, whose paths are then accessible to derivation builders [^1]. -However using the builtin features directly can be very tricky. +However, using the builtin features directly can be very tricky. [^1]: Technically only Nix store paths from the derivations inputs can be accessed, but in practice this distinction is not important. @@ -20,10 +20,10 @@ but an easier and safer interface. ## Basics The file set library is based on the concept of _file sets_, -a conceptual data type representing a collection of local files. +a data type representing a collection of local files. File sets can be created, composed and used with the various functions of the library. -The easiest way to experiment with the file set library is to first use it through `nix repl`. +The easiest way to experiment with the library is to use it through `nix repl`. ```shell-session $ nix repl -f channel:nixos-unstable @@ -37,27 +37,41 @@ which pretty-prints the files included in a given file set: ```shell-session nix-repl> fs.trace ./. null -trace: /home/user/select (all files in directory) +trace: /home/user (all files in directory) null ``` -You might wonder where the file set here is, because we just passed a _path_ to the function! +You might wonder where the file set here is, because we just passed a [_path_](https://nixos.org/manual/nix/stable/language/values#type-path) to the function! -The key is that for all functions that expect a file set for an argument, they also accepts paths instead. -Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) -to file sets containing _all files under the given path_. -We can see this from the trace `/home/user/select (all files in directory)` +The key is that for all functions that expect a file set for an argument, they _also_ accepts paths. +Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion). +The resulting file sets contain _all_ files under the given path. +We can see this from the trace `/home/user (all files in directory)` -File sets _never_ add their files to the Nix store unless explicitly requested. +Even though file sets conceptually contain local files, +they _never_ add these files to the Nix store unless explicitly requested. +So even though we pretty-printed all files in your home directory, none of its contained files were imported because of that. +This is also evident from the expression evaluating instantly. So you don't have to worry about accidentally copying secrets into the store. +:::{note} +This is in contrast to coercion of paths to strings like in `"${./.}"`, +which _does_ add all of its contained files to the Nix store! +::: + +:::{warning} +With current experimental Flakes, +the local files always get copied into the Nix store +unless you use it within a Git repository! +::: + :::{tip} The `trace` function pretty-prints its first agument and returns its second argument. But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: ```shell-session nix-repl> fs.trace ./. -trace: /home/user/select (all files in directory) +trace: /home/user (all files in directory) «lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8» ``` ::: @@ -65,27 +79,29 @@ trace: /home/user/select (all files in directory) This implicit coercion also works for files: ```shell-session -nix-repl> fs.trace ./string.txt -trace: /home/user/select -trace: - string.txt (regular) +nix-repl> fs.trace /etc/nix/nix.conf +trace: /etc/nix +trace: - nix.conf (symlink) ``` We can see that in addition to the included file, it also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). -And if we make a typo for a path that doesn't exist, the file set library adequately complains about it: +But if we make a typo for a path that doesn't exist, +the library adequately complains about it: ```shell-session -nix-repl> fs.trace ./string.nix -error: lib.fileset.trace: Argument (/home/user/select/string.nix) +nix-repl> fs.trace /etc/nix/nix.nix +error: lib.fileset.trace: Argument (/etc/nix/nix.nix) is a path that does not exist. ``` -## Setting up a local experiment +## A local directory -To further experiment with the library, let's set up a local project. +To further experiment with the library, let's set up a local directory. +To start out, create a new directory, enter it, +and set up `niv` to pin the Nixpkgs dependency: -To start out, create a new directory, enter it, and set up `niv` to pin the Nixpkgs dependency: ```shell-session $ mkdir select $ cd select @@ -96,15 +112,16 @@ $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixo For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. ::: -Then create a `default.nix` file with these contents: -```nix +Then create a `default.nix` file: + +```{code-block} nix +:caption: default.nix { system ? builtins.currentSystem, sources ? import ./nix/sources.nix, }: let pkgs = import sources.nixpkgs { - # Ensure purity config = { }; overlays = [ ]; inherit system; @@ -117,7 +134,8 @@ From now on we'll just change the contents of `package.nix` while keeping `defau For now, let's have a simple `package.nix` to verify everything works so far: -```nix +```{code-block} nix +:caption: package.nix { runCommand }: runCommand "hello" { } '' echo hello world @@ -125,6 +143,7 @@ runCommand "hello" { } '' ``` And try it out: + ```shell-session $ nix-build this derivation will be built: @@ -149,7 +168,8 @@ added to the Nix store rooted at a specific path. Let's try it out by defining `package.nix` as follows: -```nix +```{code-block} nix +:caption: package.nix { stdenv, lib }: let fs = lib.fileset; @@ -182,7 +202,9 @@ To show some more motivation, let's first use the file set library to include all files from the local directory in the build, and have it succeed by coping the `string.txt` file to the output: -```nix +```{code-block} nix +:caption: package.nix +:emphasize-lines: 4, 13-15 { stdenv, lib }: let fs = lib.fileset; @@ -213,6 +235,7 @@ this derivation will be built: ``` But if you rebuild again, you get something different! + ```shell-session $ nix-build trace: /home/user/select (all files in directory) @@ -223,21 +246,26 @@ this derivation will be built: ``` The problem here is that `nix-build` by default creates a symlink in the local directory, pointing to the result of the build: + ``` $ ls -l result result -> /nix/store/n9i6gaf80hvbfplsv7zilkbfrz47s4kn-filesets ``` In such a case, we can use the [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function. -It allows subtracting a file set from another, -resulting in a new file set that contains all files from the first argument that aren't in the second argument. +It allows "subtracting" one file set from another, +resulting in a new file set that contains all files from the first argument +that aren't in the second argument. Let's use it to filter out `./result` by changing the `sourceFiles` definition: -```nix + +```{code-block} nix +:caption: package.nix sourceFiles = fs.difference ./. ./result; ``` Building it now we get: + ```shell-session $ nix-build trace: /home/user/select @@ -260,6 +288,7 @@ because we don't include `./result` anymore. Also, running the build always gives the same result, no more rebuilding necessary! If we try to remove the `./result` symlink however, we run into a problem: + ```shell-session $ rm result $ nix-build @@ -272,7 +301,8 @@ It helpfully explains to us that for files that may not exist, we should use `maybeMissing` , so let's try it: -```nix +```{code-block} nix +:caption: package.nix sourceFiles = fs.difference ./. (fs.maybeMissing ./result); ``` @@ -320,7 +350,8 @@ One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstab to create a file set containing all of the files we don't want, and removing that instead: -```nix +```{code-block} nix +:caption: package.nix sourceFiles = fs.difference ./. @@ -336,7 +367,8 @@ This also gives us the opportunity to show the [`fileFilter`](https://nixos.org/ which as the name implies, allows filtering the files in a local path. We use it to select all files whose name ends with `.nix`: -```nix +```{code-block} nix +:caption: package.nix sourceFiles = fs.difference ./. @@ -380,7 +412,8 @@ $ touch build.sh src/select.{c,h} And then create a file set from just the ones we're interested in: -```nix +```{code-block} nix +:caption: package.nix sourceFiles = fs.unions [ ./build.sh ./string.txt @@ -445,7 +478,8 @@ $ git reset src/select.o result Now we can re-use this selection of files using `gitTracked`: -```nix +```{code-block} nix +:caption: package.nix sourceFiles = fs.gitTracked ./.; ``` @@ -477,7 +511,8 @@ This is where `intersection` comes in. It allows us to create a file set consisting only of files that are in _both_ of two file sets. In this case we only want files that are both tracked by git, and included in our exclusive selection: -```nix +```{code-block} nix +:caption: package.nix sourceFiles = fs.intersection (fs.gitTracked ./.) From 436b4aa1b9323439cc4cd28a913d0ca4b96da6f7 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Tue, 21 Nov 2023 18:29:32 +0100 Subject: [PATCH 10/50] Briefly mention problems with builtin features --- source/tutorials/file-sets.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index aa785c2627..3b1a6f868f 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -8,13 +8,23 @@ it won't have access to the local project files by default. To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store, whose paths are then accessible to derivation builders [^1]. -However, using the builtin features directly can be very tricky. [^1]: Technically only Nix store paths from the derivations inputs can be accessed, but in practice this distinction is not important. +Using these features directly can be tricky however: + +- Coercion of paths to strings, such as the wide-spread pattern of `src = ./.`, + makes the derivation dependent on the name of the current directory. + It also doesn't allow being more precise about which files to use. + +- The [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) function + (and equivalently [`lib.sources.cleanSourceWith`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.sources.cleanSourceWith)) + can address these problems. + However, it's hard to get the desired path selection using the `filter` interface. + In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. -It abstracts over the builtin features with essentially the same functionality, +It abstracts over these functions with essentially the same functionality, but an easier and safer interface. ## Basics From f8427e53285946f8f0d1529eeae831fb080d171b Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Tue, 21 Nov 2023 19:16:13 +0100 Subject: [PATCH 11/50] Minor change --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 3b1a6f868f..92b064e875 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -21,7 +21,7 @@ Using these features directly can be tricky however: - The [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) function (and equivalently [`lib.sources.cleanSourceWith`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.sources.cleanSourceWith)) can address these problems. - However, it's hard to get the desired path selection using the `filter` interface. + However, it's hard to get the desired path selection using function-based interface. In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. It abstracts over these functions with essentially the same functionality, From bd3a71f64f3a84887be483b9e2eb7690b3f5a30f Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Tue, 21 Nov 2023 21:48:04 +0100 Subject: [PATCH 12/50] Update source/tutorials/file-sets.md --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 92b064e875..0d8031c1b0 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -21,7 +21,7 @@ Using these features directly can be tricky however: - The [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) function (and equivalently [`lib.sources.cleanSourceWith`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.sources.cleanSourceWith)) can address these problems. - However, it's hard to get the desired path selection using function-based interface. + However, it's hard to get the desired path selection using the `filter` function interface. In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. It abstracts over these functions with essentially the same functionality, From 17e859e53bfb374c514bc5d9762c902422a511bc Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:11:56 +0100 Subject: [PATCH 13/50] more informative title --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 0d8031c1b0..788c6ce7de 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -1,5 +1,5 @@ (file-sets)= -# File sets +# Working with local files To build a local project in a Nix derivation, its source files must be accessible to the builder. From 285fd17c1d1c63b9adc51ba37565e4f11093ade5 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:13:14 +0100 Subject: [PATCH 14/50] shorten introduction - link to reference documentation on the `builder` executable - since the sandbox is enabled by default, we can just say that --- source/tutorials/file-sets.md | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 788c6ce7de..576b568eda 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -2,15 +2,9 @@ # Working with local files -To build a local project in a Nix derivation, its source files must be accessible to the builder. -But since the builder runs in an isolated environment (if the [sandbox](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) is enabled), -it won't have access to the local project files by default. - -To make this work regardless, the Nix language has certain builtin features to copy local paths to the Nix store, -whose paths are then accessible to derivation builders [^1]. - -[^1]: Technically only Nix store paths from the derivations inputs can be accessed, -but in practice this distinction is not important. +To build a local project in a Nix derivation, its source files must be accessible to the [`builder` executable](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). +Since by default the `builder` runs in an isolated environment that only allows reading from the Nix store, +the Nix language has built-in features to copy local files to the store and expose the resulting store paths. Using these features directly can be tricky however: From 6a61be98111f60476e46ff48d0788d50e5b74fb1 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:15:08 +0100 Subject: [PATCH 15/50] clarify wording --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 576b568eda..8f864fa037 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -15,7 +15,7 @@ Using these features directly can be tricky however: - The [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) function (and equivalently [`lib.sources.cleanSourceWith`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.sources.cleanSourceWith)) can address these problems. - However, it's hard to get the desired path selection using the `filter` function interface. + However, it's often hard to express the desired path selection using the `filter` function interface. In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. It abstracts over these functions with essentially the same functionality, From 7e7c6d79bb7c435889122a67b75a1e4bed07df9f Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:15:48 +0100 Subject: [PATCH 16/50] reword learning objective - sharpen the value proposition of the library - this meshes with and expands on the new title --- source/tutorials/file-sets.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 8f864fa037..fec4dfe98d 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -17,9 +17,8 @@ Using these features directly can be tricky however: can address these problems. However, it's often hard to express the desired path selection using the `filter` function interface. -In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset) instead. -It abstracts over these functions with essentially the same functionality, -but an easier and safer interface. +In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. +It abstracts over built-in functionality and offers a safer and more convenient interface. ## Basics From 8402ecf299b4a0ee325c4a9547ee6f544c3d0675 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:17:18 +0100 Subject: [PATCH 17/50] Basics -> File sets since the title is now different, we can be more specific on what the first section is about: file sets, generally. --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index fec4dfe98d..268b38e079 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -20,7 +20,7 @@ Using these features directly can be tricky however: In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. It abstracts over built-in functionality and offers a safer and more convenient interface. -## Basics +## File sets The file set library is based on the concept of _file sets_, a data type representing a collection of local files. From 06caaae956d268ac96fd078711d31d6ccaeaf1cf Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:18:17 +0100 Subject: [PATCH 18/50] clarify wording --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 268b38e079..e9ef4a5bd2 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -24,7 +24,7 @@ It abstracts over built-in functionality and offers a safer and more convenient The file set library is based on the concept of _file sets_, a data type representing a collection of local files. -File sets can be created, composed and used with the various functions of the library. +File sets can be created, composed, and manipulated with the various functions of the library. The easiest way to experiment with the library is to use it through `nix repl`. From d388d6d633c1b556a87e3581f25570df9bc69993 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:18:34 +0100 Subject: [PATCH 19/50] add link to command reference --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index e9ef4a5bd2..af42e4328e 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -26,7 +26,7 @@ The file set library is based on the concept of _file sets_, a data type representing a collection of local files. File sets can be created, composed, and manipulated with the various functions of the library. -The easiest way to experiment with the library is to use it through `nix repl`. +The easiest way to experiment with the library is to use it through [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): ```shell-session $ nix repl -f channel:nixos-unstable From 76905abea74b49efa6f72a6a4ab17358cb614158 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:19:20 +0100 Subject: [PATCH 20/50] shorten introduction to `trace` --- source/tutorials/file-sets.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index af42e4328e..6621997579 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -34,9 +34,7 @@ $ nix repl -f channel:nixos-unstable nix-repl> fs = lib.fileset ``` -It's probably the easiest to just jump right in -by using the [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function, -which pretty-prints the files included in a given file set: +The [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function pretty-prints the files included in a given file set: ```shell-session nix-repl> fs.trace ./. null From 9fd9700c883a7f75be2acb27a1c989ae791661f8 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:20:24 +0100 Subject: [PATCH 21/50] use log ellipsis as in the other tutorials brackets in console output invoke weird associations, as they could be syntactically relevant. bare ellipses are unusual enough one can expect them to be intuitively parsed as artificial. --- source/tutorials/file-sets.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 6621997579..2bd12c2bc2 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -30,7 +30,7 @@ The easiest way to experiment with the library is to use it through [`nix repl`] ```shell-session $ nix repl -f channel:nixos-unstable -[...] +... nix-repl> fs = lib.fileset ``` @@ -231,7 +231,7 @@ $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: /nix/store/d8mj3z49s24q46rncma6v9kvi6xbx4vq-filesets.drv -[...] +... /nix/store/v5sx60xd9lvylgqcyqpchac1a2k8300c-filesets ``` @@ -242,7 +242,7 @@ $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: /nix/store/1xb412x3fzavr8d8c3hbl3fv9kyvj77c-filesets.drv -[...] +... /nix/store/n9i6gaf80hvbfplsv7zilkbfrz47s4kn-filesets ``` @@ -276,11 +276,11 @@ trace: - package.nix (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/7960rh64d4zlkspmf4h51g4zys3lcjyj-filesets.drv -[...] +... /nix/store/aicvbzjvqzn06nbgpbrwqi47rxqdiqv9-filesets $ nix-build -[...] +... /nix/store/aicvbzjvqzn06nbgpbrwqi47rxqdiqv9-filesets ``` @@ -314,7 +314,7 @@ $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: /nix/store/ygpx17kshzc6bj3c71xlda8szw6qi1sr-filesets.drv -[...] +... /nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets $ nix-build @@ -343,7 +343,7 @@ trace: - package.nix (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/zmgpqlpfz2jq0w9rdacsnpx8ni4n77cn-filesets.drv -[...] +... /nix/store/6pffjljjy3c7kla60nljk3fad4q4kkzn-filesets ``` @@ -441,7 +441,7 @@ trace: - string.txt (regular) this derivation will be built: /nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv building '/nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv'... -[...] +... /nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets $ touch src/select.o @@ -500,7 +500,7 @@ trace: - select.h (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/vn21azx8y06cjq80lrvib8ia4xxpwn3d-filesets.drv -[...] +... /nix/store/4xdfxm910x1i2qapv49caiibymfjhvla-filesets ``` From 5ca3739c36eb895e9b007acb90cbebb67ad4442d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:23:05 +0100 Subject: [PATCH 22/50] shorten introduction to path arguments --- source/tutorials/file-sets.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 2bd12c2bc2..dc3984532b 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -42,12 +42,9 @@ trace: /home/user (all files in directory) null ``` -You might wonder where the file set here is, because we just passed a [_path_](https://nixos.org/manual/nix/stable/language/values#type-path) to the function! - -The key is that for all functions that expect a file set for an argument, they _also_ accepts paths. -Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion). -The resulting file sets contain _all_ files under the given path. -We can see this from the trace `/home/user (all files in directory)` +All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path). +Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion), and the resulting file sets contain _all_ files under the given path. +In the previous trace this is indicated by `(all files in directory)`. Even though file sets conceptually contain local files, they _never_ add these files to the Nix store unless explicitly requested. From 568925ec8f01e237beb485f5fbfacd7b84ad5cd6 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:26:38 +0100 Subject: [PATCH 23/50] move tip before the other notes three highlight boxes in a row seem a bit too much, also the tip is relevant immediately after the first example. --- source/tutorials/file-sets.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index dc3984532b..26dbbc48d8 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -46,6 +46,17 @@ All functions that expect a file set for an argument also accept a [path](https: Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion), and the resulting file sets contain _all_ files under the given path. In the previous trace this is indicated by `(all files in directory)`. +:::{tip} +The `trace` function pretty-prints its first agument and returns its second argument. +But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: + +```shell-session +nix-repl> fs.trace ./. +trace: /home/user (all files in directory) +«lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8» +``` +::: + Even though file sets conceptually contain local files, they _never_ add these files to the Nix store unless explicitly requested. So even though we pretty-printed all files in your home directory, none of its contained files were imported because of that. @@ -63,17 +74,6 @@ the local files always get copied into the Nix store unless you use it within a Git repository! ::: -:::{tip} -The `trace` function pretty-prints its first agument and returns its second argument. -But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: - -```shell-session -nix-repl> fs.trace ./. -trace: /home/user (all files in directory) -«lambda @ /nix/store/1czr278x24s3bl6qdnifpvm5z03wfi2p-nixpkgs-src/lib/fileset/default.nix:555:8» -``` -::: - This implicit coercion also works for files: ```shell-session From 97bbe4fcf7afd3cd2c98339acded1f60c4615301 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:31:40 +0100 Subject: [PATCH 24/50] restructure notes on copying files --- source/tutorials/file-sets.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 26dbbc48d8..bc5ca0e251 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -57,15 +57,15 @@ trace: /home/user (all files in directory) ``` ::: -Even though file sets conceptually contain local files, -they _never_ add these files to the Nix store unless explicitly requested. -So even though we pretty-printed all files in your home directory, none of its contained files were imported because of that. +Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested. +You don't have to worry about accidentally copying secrets into the world-readable store. + +In this example, although we pretty-printed the home directory, no files were copied. This is also evident from the expression evaluating instantly. -So you don't have to worry about accidentally copying secrets into the store. :::{note} -This is in contrast to coercion of paths to strings like in `"${./.}"`, -which _does_ add all of its contained files to the Nix store! +This is in contrast to coercion of paths to strings such as in `"${./.}"`, +which copies the whole directory to the Nix store on evaluation! ::: :::{warning} From fb6f6302d04c709d978abb282babc6b402cb573d Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:32:36 +0100 Subject: [PATCH 25/50] add link to flakes documentation, add emphasis --- source/tutorials/file-sets.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index bc5ca0e251..5c359aad51 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -69,9 +69,7 @@ which copies the whole directory to the Nix store on evaluation! ::: :::{warning} -With current experimental Flakes, -the local files always get copied into the Nix store -unless you use it within a Git repository! +With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! ::: This implicit coercion also works for files: From 767155bc15455637bd65bff2a87d092a9ac633fa Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:33:05 +0100 Subject: [PATCH 26/50] trim down phrasing --- source/tutorials/file-sets.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 5c359aad51..964c4cd442 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -80,11 +80,9 @@ trace: /etc/nix trace: - nix.conf (symlink) ``` -We can see that in addition to the included file, -it also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). +In addition to the included file, this also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). -But if we make a typo for a path that doesn't exist, -the library adequately complains about it: +If a given path doesn't exist, the library will complain: ```shell-session nix-repl> fs.trace /etc/nix/nix.nix From 152953ef302a6fd58bb5096c1b6ceccce9dd454b Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:34:52 +0100 Subject: [PATCH 27/50] A local directory -> Example project - more telling section title - use imperative in instructions --- source/tutorials/file-sets.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 964c4cd442..c0374de995 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -90,10 +90,10 @@ error: lib.fileset.trace: Argument (/etc/nix/nix.nix) is a path that does not exist. ``` -## A local directory +## Example project -To further experiment with the library, let's set up a local directory. -To start out, create a new directory, enter it, +To further experiment with the library, make a sample project. +Create a new directory, enter it, and set up `niv` to pin the Nixpkgs dependency: ```shell-session From 51a78a0bd4cc8c8f9126cf8e7ae54a8ecbdd5428 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:36:32 +0100 Subject: [PATCH 28/50] channel -> channel branch this follows the convention established in the FAQ on releases --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index c0374de995..4abdc99db3 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -103,7 +103,7 @@ $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixo ``` :::{note} -For now we're using the nixos-unstable channel, since no stable channel has all the features we need yet. +We're using the `nixos-unstable` channel branch here, since no stable release has all the features needed for this tutorial. ::: Then create a `default.nix` file: From a6e5f2675f59c762fb78cb26aa98ccf78bd75dd8 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 01:38:01 +0100 Subject: [PATCH 29/50] package.nix -> build.nix this follows the convention established in other tutorials. also arguably, this file is *not* the package per se, but simply contains a representation of build instructions. --- source/tutorials/file-sets.md | 42 +++++++++++++++++------------------ 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 4abdc99db3..9065a84c07 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -121,15 +121,15 @@ let inherit system; }; in -pkgs.callPackage ./package.nix { } +pkgs.callPackage ./build.nix { } ``` -From now on we'll just change the contents of `package.nix` while keeping `default.nix` the same. +From now on we'll just change the contents of `build.nix` while keeping `default.nix` the same. -For now, let's have a simple `package.nix` to verify everything works so far: +For now, let's have a simple `build.nix` to verify everything works so far: ```{code-block} nix -:caption: package.nix +:caption: build.nix { runCommand }: runCommand "hello" { } '' echo hello world @@ -160,10 +160,10 @@ That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-li It allows us to create a Nix store path containing exactly only the files that are in the file set, added to the Nix store rooted at a specific path. -Let's try it out by defining `package.nix` as follows: +Let's try it out by defining `build.nix` as follows: ```{code-block} nix -:caption: package.nix +:caption: build.nix { stdenv, lib }: let fs = lib.fileset; @@ -197,7 +197,7 @@ let's first use the file set library to include all files from the local directo and have it succeed by coping the `string.txt` file to the output: ```{code-block} nix -:caption: package.nix +:caption: build.nix :emphasize-lines: 4, 13-15 { stdenv, lib }: let @@ -254,7 +254,7 @@ that aren't in the second argument. Let's use it to filter out `./result` by changing the `sourceFiles` definition: ```{code-block} nix -:caption: package.nix +:caption: build.nix sourceFiles = fs.difference ./. ./result; ``` @@ -265,7 +265,7 @@ $ nix-build trace: /home/user/select trace: - default.nix (regular) trace: - nix (all files in directory) -trace: - package.nix (regular) +trace: - build.nix (regular) trace: - string.txt (regular) this derivation will be built: /nix/store/7960rh64d4zlkspmf4h51g4zys3lcjyj-filesets.drv @@ -296,7 +296,7 @@ we should use `maybeMissing` , -so let's try it: +Follow the instructions in the error message, and use `maybeMissing` to create a file set from a path that may not exist (in which case the file set will be empty): -```{code-block} nix + + +```{code-block} diff :caption: build.nix - sourceFiles = fs.difference ./. (fs.maybeMissing ./result); + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = fs.difference ./. ./result; ++ sourceFiles = fs.difference ./. (fs.maybeMissing ./result); + in ``` -This now works, reliably filtering out `./result` if it exists: +This now works, using the whole directory since `./result` is not present: ``` $ nix-build trace: /home/user/select (all files in directory) this derivation will be built: - /nix/store/ygpx17kshzc6bj3c71xlda8szw6qi1sr-filesets.drv + /nix/store/zr19bv51085zz005yk7pw4s9sglmafvn-fileset.drv ... -/nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets +/nix/store/vhyhk6ij39gjapqavz1j1x3zbiy3qc1a-fileset +``` + +Another build attempt will produce a different trace, but the same output path: +``` $ nix-build -trace: /home/user/select +trace: /home/user/fileset +trace: - build.nix (regular) trace: - default.nix (regular) +trace: - hello.txt (regular) trace: - nix (all files in directory) -trace: - build.nix (regular) -trace: - string.txt (regular) -/nix/store/bzvhlr9h2zwqi7rr9i1j193z9hkskhmk-filesets +trace: - world.txt (regular) +/nix/store/vhyhk6ij39gjapqavz1j1x3zbiy3qc1a-fileset ``` -## Adding file sets together +## Union (exclude) + +There is still a problem: +Changing _any_ of the included files causes the derivation to be rebuilt, even though it doesn't depend on those files. -We still have a problem however: -Changing _any_ of the included files causes the derivation to be rebuilt, -even though it doesn't depend on those files. +Append an empty line to `build.nix`: ```shell-session -$ echo "# Just a comment" >> build.nix +$ echo >> build.nix +``` + +Again, Nix will start from scratch: +```shell-session $ nix-build trace: /home/user/select trace: - default.nix (regular) @@ -346,77 +439,104 @@ this derivation will be built: /nix/store/6pffjljjy3c7kla60nljk3fad4q4kkzn-filesets ``` -One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions) -to create a file set containing all of the files we don't want, -and removing that instead: +One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.unions). -```{code-block} nix +Create a file set containing a union of the files to exclude (`fs.unions [ ... ]`), and subtract it (`difference`) from the complete directory (`./.`): + +```{code-block} diff :caption: build.nix - sourceFiles = - fs.difference - ./. - (fs.unions [ - (fs.maybeMissing ./result) - ./default.nix - ./build.nix - ./nix - ]); + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = fs.difference ./. (fs.maybeMissing ./result); ++ sourceFiles = ++ fs.difference ++ ./. ++ (fs.unions [ ++ (fs.maybeMissing ./result) ++ ./default.nix ++ ./build.nix ++ ./nix ++ ]); + in ``` -This also gives us the opportunity to show the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function, -which as the name implies, allows filtering the files in a local path. -We use it to select all files whose name ends with `.nix`: +Changing any of the excluded files now doesn't necessarily require a rebuild anymore. -```{code-block} nix -:caption: build.nix - sourceFiles = - fs.difference - ./. - (fs.unions [ - (fs.maybeMissing ./result) - (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) - ./nix - ]); -``` +Check it and modify one of the excluded files again: -Changing any of the removed files now doesn't necessarily require a rebuild anymore: +``` +$ echo >> build.nix +``` -```shell-session +``` $ nix-build -trace: /home/user/select -trace: - string.txt (regular) -/nix/store/clrd19vn5cv6n7x7hzajq1fv43qig7cp-filesets +trace: /home/user/fileset +trace: - hello.txt (regular) +trace: - world.txt (regular) +/nix/store/ckn40y7hgqphhbhyrq64h9r6rvdh973r-fileset +``` + +## Filter + +Dealing with a large number of files, independent of their location, can be done programmatically with the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function. -$ echo "# Just a comment" >> build.nix +Use it to select all files with a name ending in `.nix`: +```{code-block} diff +:caption: build.nix + { stdenv, lib }: + let + fs = lib.fileset; + sourceFiles = + fs.difference + ./. + (fs.unions [ + (fs.maybeMissing ./result) +- ./default.nix +- ./build.nix ++ (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) + ./nix + ]); + in +``` + +This does not change the result: + +```shell-session $ nix-build -trace: /home/user/select -trace: - string.txt (regular) -/nix/store/clrd19vn5cv6n7x7hzajq1fv43qig7cp-filesets +trace: /home/user/fileset +trace: - hello.txt (regular) +trace: - world.txt (regular) +/nix/store/ckn40y7hgqphhbhyrq64h9r6rvdh973r-fileset ``` -Notable with this approach is that new files added to the current directory are _included_ by default. +Notably, with this approach, new files added to the source directory are _included_ by default. Depending on your project, this might be a better fit than the alternative in the next section. -## Only including necessary files +## Union (include) -To contrast the above approach, we can also directly use `unions` to select only the files we want to _include_. -This means that new files added to the current directory would be _excluded_ by default. +To contrast the previous approach, `unions` can also be used to select only the files to _include_. +This means that new files added to the current directory would be ignored by default. -To demonstrate, let's create some extra files to select: +Create some additional files: ```shell-session $ mkdir src $ touch build.sh src/select.{c,h} ``` -And then create a file set from just the ones we're interested in: +Then create a file set from only the files to be included explicitly: ```{code-block} nix :caption: build.nix +{ stdenv, lib }: +let + fs = lib.fileset; sourceFiles = fs.unions [ + ./hello.txt + ./world.txt ./build.sh - ./string.txt (fs.fileFilter (file: lib.hasSuffix ".c" file.name @@ -425,49 +545,72 @@ And then create a file set from just the ones we're interested in: ./src ) ]; +in + +fs.trace sourceFiles + +stdenv.mkDerivation { + name = "fileset"; + src = fs.toSource { + root = ./.; + fileset = sourceFiles; + }; + postInstall = '' + cp -vr $src $out + ''; +} ``` -When building this you'll see that only the specified files are used, even when a new one is added: +The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately: ```shell-session $ nix-build -trace: /home/user/select +trace: /home/user/fileset trace: - build.sh (regular) -trace: - src -trace: - select.c (regular) -trace: - select.h (regular) -trace: - string.txt (regular) +trace: - hello.txt (regular) +trace: - src (all files in directory) +trace: - world.txt (regular) this derivation will be built: - /nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv -building '/nix/store/gzj9j9dk2qyd46y1g2wkpkrbc3f2nm5g-filesets.drv'... + /nix/store/sjzkn07d6a4qfp60p6dc64pzvmmdafff-fileset.drv ... -/nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source' -> '/nix/store/zl4n1g6is4cmsqf02 +dci5b2h5zd0ia4r-fileset' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/build.sh' -> '/nix/store/zl4n1g6i +s4cmsqf02dci5b2h5zd0ia4r-fileset/build.sh' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/hello.txt' -> '/nix/store/zl4n1g6 +is4cmsqf02dci5b2h5zd0ia4r-fileset/hello.txt' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/world.txt' -> '/nix/store/zl4n1g6 +is4cmsqf02dci5b2h5zd0ia4r-fileset/world.txt' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src' -> '/nix/store/zl4n1g6is4cms +qf02dci5b2h5zd0ia4r-fileset/src' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.c' -> '/nix/store/zl4n +1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.c' +'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.h' -> '/nix/store/zl4n +1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.h' +... +/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset +``` +Only the specified files are used, even when a new one is added: + +```shell-session $ touch src/select.o $ nix-build -trace: /home/user/select trace: - build.sh (regular) +trace: - hello.txt (regular) trace: - src trace: - select.c (regular) trace: - select.h (regular) -trace: - string.txt (regular) -/nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets +trace: - world.txt (regular) +/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset ``` -## Git +## Matching files tracked by Git -In case we track files with Git, we can use [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) to re-use the same set of files by Git. +If a directory is part of a Git repository, [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) automatically filters for files that are tracked by Git. -:::{note} -With current experimental Flakes, -it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, -even with `nix build path:.`. -However it's also not needed, because by default, -`nix build` only allows access to Git-tracked files. -::: - -Let's create a local Git repository and add track all files except `src/select.o` and `./result` to it: +Create a local Git repository and add all files except `src/select.o` and `./result` to it: ```shell-session $ git init @@ -488,28 +631,40 @@ Building we get ```shell-session $ nix-build warning: Git tree '/home/user/select' is dirty -trace: /home/user/select +trace: /home/vg/src/nix.dev/fileset +trace: - build.nix (regular) trace: - build.sh (regular) trace: - default.nix (regular) +trace: - hello.txt (regular) trace: - nix (all files in directory) -trace: - build.nix (regular) trace: - src trace: - select.c (regular) trace: - select.h (regular) -trace: - string.txt (regular) +trace: - world.txt (regular) this derivation will be built: - /nix/store/vn21azx8y06cjq80lrvib8ia4xxpwn3d-filesets.drv + /nix/store/p9aw3fl5xcjbgg9yagykywvskzgrmk5y-fileset.drv ... -/nix/store/4xdfxm910x1i2qapv49caiibymfjhvla-filesets +/nix/store/cw4bza1r27iimzrdbfl4yn5xr36d6k5l-fileset ``` -This includes too much though, we don't need all of these files to build the derivation. +This includes too much though, as not all of these files are needed to build the derivation as originally intended. + +:::{note} +With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with + +```shell-session +nix build path:. +``` + +However it's also not needed, because by default, `nix build` only allows access to files tracked by Git. +::: ## Intersection This is where `intersection` comes in. -It allows us to create a file set consisting only of files that are in _both_ of two file sets. -In this case we only want files that are both tracked by git, and included in our exclusive selection: +It allows creating a file set that consists only of files that are in _both_ of the two given file sets. + +Select all files that are both tracked by Git *and* relevant for the build: ```{code-block} nix :caption: build.nix @@ -517,24 +672,25 @@ In this case we only want files that are both tracked by git, and included in ou fs.intersection (fs.gitTracked ./.) (fs.unions [ + ./hello.txt + ./world.txt ./build.sh - ./string.txt ./src ]); ``` -At last we get what we expect: +This will produce the same output as in the other approach and therefore re-use a previous build result: ```shell-session $ nix-build warning: Git tree '/home/user/select' is dirty -trace: /home/user/select trace: - build.sh (regular) +trace: - hello.txt (regular) trace: - src trace: - select.c (regular) trace: - select.h (regular) -trace: - string.txt (regular) -/nix/store/sb4g8skwvpwbay5kdpnyhwjglxqzim28-filesets +trace: - world.txt (regular) +/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset ``` ## Conclusion From fa037e69f6237c1d3a9fcce8bf39f818e445ce9e Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 22 Nov 2023 05:19:27 +0100 Subject: [PATCH 34/50] fix omission Co-authored-by: Silvan Mosberger --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index f6a41f8dc8..41753f58c5 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -203,7 +203,7 @@ t.drv'. That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in: It adds exactly the files in the given file set to a directory in the Nix store, starting from a specified root path. -Define `package.nix` as follows: +Define `build.nix` as follows: ```{code-block} nix :caption: build.nix From 429bbb0621b8049b151446d2240b594d6cb3f203 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:24:47 +0100 Subject: [PATCH 35/50] Update source/tutorials/file-sets.md Co-authored-by: Valentin Gagarin --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 41753f58c5..63b2c1b997 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -413,7 +413,7 @@ trace: - world.txt (regular) /nix/store/vhyhk6ij39gjapqavz1j1x3zbiy3qc1a-fileset ``` -## Union (exclude) +## Union (explicitly exclude files) There is still a problem: Changing _any_ of the included files causes the derivation to be rebuilt, even though it doesn't depend on those files. From 16db5641264c74e3a3b72bbd4c8fc940726674e2 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:25:51 +0100 Subject: [PATCH 36/50] Update source/tutorials/file-sets.md Co-authored-by: Valentin Gagarin --- source/tutorials/file-sets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 63b2c1b997..520a342b67 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -511,7 +511,7 @@ trace: - world.txt (regular) /nix/store/ckn40y7hgqphhbhyrq64h9r6rvdh973r-fileset ``` -Notably, with this approach, new files added to the source directory are _included_ by default. +Notably, the approach of using `difference ./.` explicitly selects the files to _exclude_, which means that new files added to the source directory are included by default. Depending on your project, this might be a better fit than the alternative in the next section. ## Union (include) From 993bbc17879438c14f90ca60ede792f835adbc3c Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:30:16 +0100 Subject: [PATCH 37/50] Apply suggestions from code review --- source/tutorials/file-sets.md | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 520a342b67..276c863f8f 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -69,7 +69,7 @@ which copies the whole directory to the Nix store on evaluation! ::: :::{warning} -With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! +With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! ::: This implicit coercion also works for files: @@ -514,7 +514,7 @@ trace: - world.txt (regular) Notably, the approach of using `difference ./.` explicitly selects the files to _exclude_, which means that new files added to the source directory are included by default. Depending on your project, this might be a better fit than the alternative in the next section. -## Union (include) +## Union (explicitly include files) To contrast the previous approach, `unions` can also be used to select only the files to _include_. This means that new files added to the current directory would be ignored by default. @@ -556,7 +556,7 @@ stdenv.mkDerivation { fileset = sourceFiles; }; postInstall = '' - cp -vr $src $out + cp -vr . $out ''; } ``` @@ -573,20 +573,13 @@ trace: - world.txt (regular) this derivation will be built: /nix/store/sjzkn07d6a4qfp60p6dc64pzvmmdafff-fileset.drv ... -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source' -> '/nix/store/zl4n1g6is4cmsqf02 -dci5b2h5zd0ia4r-fileset' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/build.sh' -> '/nix/store/zl4n1g6i -s4cmsqf02dci5b2h5zd0ia4r-fileset/build.sh' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/hello.txt' -> '/nix/store/zl4n1g6 -is4cmsqf02dci5b2h5zd0ia4r-fileset/hello.txt' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/world.txt' -> '/nix/store/zl4n1g6 -is4cmsqf02dci5b2h5zd0ia4r-fileset/world.txt' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src' -> '/nix/store/zl4n1g6is4cms -qf02dci5b2h5zd0ia4r-fileset/src' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.c' -> '/nix/store/zl4n -1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.c' -'/nix/store/6k6pv78b1dkdhbdlzxar5vb3z3c8fwza-source/src/select.h' -> '/nix/store/zl4n -1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.h' +'.' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset' +'./build.sh' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/build.sh' +'./hello.txt' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/hello.txt' +'./world.txt' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/world.txt' +'./src' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src' +'./src/select.c' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.c' +'./src/select.h' -> '/nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset/src/select.h' ... /nix/store/zl4n1g6is4cmsqf02dci5b2h5zd0ia4r-fileset ``` @@ -650,7 +643,7 @@ this derivation will be built: This includes too much though, as not all of these files are needed to build the derivation as originally intended. :::{note} -With the [`flakes` experimental feature](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake) enabled, it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with +With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with ```shell-session nix build path:. From a2ac364b8cf813cc6cc3f1375f49b1411e038092 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 05:35:23 +0100 Subject: [PATCH 38/50] Add Git to Vale's vocab --- vale/Vocab/Nix/accept.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/vale/Vocab/Nix/accept.txt b/vale/Vocab/Nix/accept.txt index 4f8d3572ac..fa0b9ea5a0 100644 --- a/vale/Vocab/Nix/accept.txt +++ b/vale/Vocab/Nix/accept.txt @@ -3,6 +3,7 @@ Docker Dolstra Eelco Frequently Asked Questions +Git GitHub GitHub Actions GitHub Actions Cache From bc00f00c5c81fde43d617a0847df222af52bfe32 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 06:35:26 +0100 Subject: [PATCH 39/50] More consistency in using diffs and prompts --- source/tutorials/file-sets.md | 119 +++++++++++++++++++++------------- 1 file changed, 75 insertions(+), 44 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 276c863f8f..0cde12b391 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -127,8 +127,8 @@ pkgs.callPackage ./build.nix { } Add two source files to work with: ```shell-session -echo hello > hello.txt -echo world > world.txt +$ echo hello > hello.txt +$ echo world > world.txt ``` Start with a minimal `build.nix` based on `stdenv.mkDerivation`: @@ -146,7 +146,7 @@ stdenv.mkDerivation { } ``` -Try it out: +Try it out (note that it will take some time to fetch the first time around): ```shell-session $ nix-build @@ -528,37 +528,47 @@ $ touch build.sh src/select.{c,h} Then create a file set from only the files to be included explicitly: -```{code-block} nix +```{code-block} diff :caption: build.nix -{ stdenv, lib }: -let - fs = lib.fileset; - sourceFiles = fs.unions [ - ./hello.txt - ./world.txt - ./build.sh - (fs.fileFilter - (file: - lib.hasSuffix ".c" file.name - || lib.hasSuffix ".h" file.name - ) - ./src - ) - ]; -in + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = +- fs.difference +- ./. +- (fs.unions [ +- (fs.maybeMissing ./result) +- (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) +- ./nix +- ]); ++ sourceFiles = fs.unions [ ++ ./hello.txt ++ ./world.txt ++ ./build.sh ++ (fs.fileFilter ++ (file: ++ lib.hasSuffix ".c" file.name ++ || lib.hasSuffix ".h" file.name ++ ) ++ ./src ++ ) ++ ]; + in -fs.trace sourceFiles + fs.trace sourceFiles -stdenv.mkDerivation { - name = "fileset"; - src = fs.toSource { - root = ./.; - fileset = sourceFiles; - }; - postInstall = '' - cp -vr . $out - ''; -} + stdenv.mkDerivation { + name = "fileset"; + src = fs.toSource { + root = ./.; + fileset = sourceFiles; + }; + postInstall = '' +- mkdir $out +- cp -v {hello,world}.txt $out ++ cp -vr . $out + ''; + } ``` The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately: @@ -614,9 +624,25 @@ $ git reset src/select.o result Now we can re-use this selection of files using `gitTracked`: -```{code-block} nix +```{code-block} diff :caption: build.nix - sourceFiles = fs.gitTracked ./.; + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = fs.unions [ +- ./hello.txt +- ./world.txt +- ./build.sh +- (fs.fileFilter +- (file: +- lib.hasSuffix ".c" file.name +- || lib.hasSuffix ".h" file.name +- ) +- ./src +- ) +- ]; ++ sourceFiles = fs.gitTracked ./.; + in ``` Building we get @@ -646,7 +672,7 @@ This includes too much though, as not all of these files are needed to build the With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with ```shell-session -nix build path:. +$ nix build path:. ``` However it's also not needed, because by default, `nix build` only allows access to files tracked by Git. @@ -659,17 +685,22 @@ It allows creating a file set that consists only of files that are in _both_ of Select all files that are both tracked by Git *and* relevant for the build: -```{code-block} nix +```{code-block} diff :caption: build.nix - sourceFiles = - fs.intersection - (fs.gitTracked ./.) - (fs.unions [ - ./hello.txt - ./world.txt - ./build.sh - ./src - ]); + { stdenv, lib }: + let + fs = lib.fileset; +- sourceFiles = fs.gitTracked ./.; ++ sourceFiles = ++ fs.intersection ++ (fs.gitTracked ./.) ++ (fs.unions [ ++ ./hello.txt ++ ./world.txt ++ ./build.sh ++ ./src ++ ]); + in ``` This will produce the same output as in the other approach and therefore re-use a previous build result: From 0d60facb042c67fd324224c60753afa0d62bdd83 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 13:38:10 +0100 Subject: [PATCH 40/50] inputs -> sources --- source/tutorials/file-sets.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 0cde12b391..2f93b3b07e 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -112,10 +112,10 @@ Then create a `default.nix` file with the following contents: :caption: default.nix { system ? builtins.currentSystem, - inputs ? import ./nix/sources.nix, + sources ? import ./nix/sources.nix, }: let - pkgs = import inputs.nixpkgs { + pkgs = import sources.nixpkgs { config = { }; overlays = [ ]; inherit system; From 3befcc49f9e1364d24c7a010f50063bf4902b74a Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 19:37:16 +0100 Subject: [PATCH 41/50] Apply suggestions from code review Co-authored-by: Robert Hensing Co-authored-by: Valentin Gagarin --- source/tutorials/file-sets.md | 30 +++++++++++------------------- 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 2f93b3b07e..2efdf54f15 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -10,14 +10,14 @@ Using these features directly can be tricky however: - Coercion of paths to strings, such as the wide-spread pattern of `src = ./.`, makes the derivation dependent on the name of the current directory. - It also doesn't allow being more precise about which files to use. + It always adds the entire directory, including unneeded files that cause unnecessary new builds when they change. - The [`builtins.path`](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-path) function (and equivalently [`lib.sources.cleanSourceWith`](https://nixos.org/manual/nixpkgs/stable/#function-library-lib.sources.cleanSourceWith)) can address these problems. However, it's often hard to express the desired path selection using the `filter` function interface. -In this tutorial you'll learn how to use the [file set library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. +In this tutorial you'll learn how to use the Nixpkgs [`fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. It abstracts over built-in functionality and offers a safer and more convenient interface. ## File sets @@ -26,7 +26,7 @@ The file set library is based on the concept of _file sets_, a data type representing a collection of local files. File sets can be created, composed, and manipulated with the various functions of the library. -The easiest way to experiment with the library is to use it through [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): +You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): ```shell-session $ nix repl -f channel:nixos-unstable @@ -43,11 +43,11 @@ null ``` All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path). -Such path arguments are then [implicitly coerced](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion), and the resulting file sets contain _all_ files under the given path. +Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path. In the previous trace this is indicated by `(all files in directory)`. :::{tip} -The `trace` function pretty-prints its first agument and returns its second argument. +The `trace` function pretty-prints its first argument and returns its second argument. But since you often just need the pretty-printing in `nix repl`, you can omit the second argument: ```shell-session @@ -58,10 +58,9 @@ trace: /home/user (all files in directory) ::: Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested. -You don't have to worry about accidentally copying secrets into the world-readable store. +You don't have to worry as much about accidentally copying secrets into the world-readable store. In this example, although we pretty-printed the home directory, no files were copied. -This is also evident from the expression evaluating instantly. :::{note} This is in contrast to coercion of paths to strings such as in `"${./.}"`, @@ -82,13 +81,6 @@ trace: - nix.conf (symlink) In addition to the included file, this also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). -If a given path doesn't exist, the library will complain: - -```shell-session -nix-repl> fs.trace /etc/nix/nix.nix -error: lib.fileset.trace: Argument (/etc/nix/nix.nix) - is a path that does not exist. -``` ## Example project @@ -103,7 +95,7 @@ $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixo ``` :::{note} -We're using the `nixos-unstable` channel branch here, since no stable release has all the features needed for this tutorial. +We're using the `nixos-unstable` channel branch here, because the library is part of NixOS 23.11 and up, which has not been released at the time of writing. ::: Then create a `default.nix` file with the following contents: @@ -461,7 +453,7 @@ Create a file set containing a union of the files to exclude (`fs.unions [ ... ] in ``` -Changing any of the excluded files now doesn't necessarily require a rebuild anymore. +Changing any of the excluded files now doesn't necessarily cause a new build anymore. Check it and modify one of the excluded files again: @@ -501,7 +493,7 @@ Use it to select all files with a name ending in `.nix`: in ``` -This does not change the result: +This does not change the result, even if we add a new `.nix` file. ```shell-session $ nix-build @@ -597,7 +589,7 @@ this derivation will be built: Only the specified files are used, even when a new one is added: ```shell-session -$ touch src/select.o +$ touch src/select.o src/README.md $ nix-build trace: - build.sh (regular) @@ -720,6 +712,6 @@ trace: - world.txt (regular) ## Conclusion We have shown some examples on how to use all of the fundamental file set functions. -For more complex behavior, they can be composed as needed. +For more complex use cases, they can be composed as needed. For the complete list and more details, see the [`lib.fileset` reference documentation](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset). From 3b2dfd8fb0b74e025892047a831839597897a2c7 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 20:15:03 +0100 Subject: [PATCH 42/50] Minor changes after review --- source/tutorials/file-sets.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 2efdf54f15..22b4a74bc2 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -220,7 +220,7 @@ stdenv.mkDerivation { } ``` -The call to `fs.trace` will print the file set that will be used as a derivation input. +The call to `fs.trace` prints the file set that will be used as a derivation input. The build will succeed: @@ -408,7 +408,7 @@ trace: - world.txt (regular) ## Union (explicitly exclude files) There is still a problem: -Changing _any_ of the included files causes the derivation to be rebuilt, even though it doesn't depend on those files. +Changing _any_ of the included files causes the derivation to be built again, even though it doesn't depend on those files. Append an empty line to `build.nix`: @@ -471,7 +471,8 @@ trace: - world.txt (regular) ## Filter -Dealing with a large number of files, independent of their location, can be done programmatically with the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function. +With the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function, +you can find sets of files satisfying a criteria. Use it to select all files with a name ending in `.nix`: From b3f67ce6de7dfd62366bab9aef67da8c461265ff Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 20:18:30 +0100 Subject: [PATCH 43/50] Combine two paragraphs --- source/tutorials/file-sets.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 22b4a74bc2..afaa4db5e9 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -61,11 +61,8 @@ Even though file sets conceptually contain local files, these files are *never* You don't have to worry as much about accidentally copying secrets into the world-readable store. In this example, although we pretty-printed the home directory, no files were copied. - -:::{note} This is in contrast to coercion of paths to strings such as in `"${./.}"`, which copies the whole directory to the Nix store on evaluation! -::: :::{warning} With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! From 67b688adb6addc00a0a7af482add9460ea6acae5 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 20:20:50 +0100 Subject: [PATCH 44/50] Don't use nix.conf --- source/tutorials/file-sets.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index afaa4db5e9..55fbf001c4 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -71,9 +71,13 @@ With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-c This implicit coercion also works for files: ```shell-session -nix-repl> fs.trace /etc/nix/nix.conf -trace: /etc/nix -trace: - nix.conf (symlink) +$ touch some-file +``` + +```shell-session +nix-repl> fs.trace ./some-file +trace: /home/user +trace: - some-file (regular) ``` In addition to the included file, this also prints its [file type](https://nixos.org/manual/nix/stable/language/builtins.html#builtins-readFileType). From e4700af5e7a1b13cb2609045a34a4f580b065935 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 20:29:23 +0100 Subject: [PATCH 45/50] Update source/tutorials/file-sets.md --- source/tutorials/file-sets.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 55fbf001c4..51dab39131 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -2,7 +2,9 @@ # Working with local files -To build a local project in a Nix derivation, its source files must be accessible to the [`builder` executable](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). +To build a local project in a Nix derivation, +its source files must be [accessible](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) +to the [derivation builder](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). Since by default the `builder` runs in an isolated environment that only allows reading from the Nix store, the Nix language has built-in features to copy local files to the store and expose the resulting store paths. From 6afd231b97af63b8b2d61a8f5b63bbd5825d3b40 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 20:44:26 +0100 Subject: [PATCH 46/50] Some updates Use master branch (nixos-unstable doesn't have maybeMissing yet) --- source/tutorials/file-sets.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 51dab39131..5d39227f2d 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -1,6 +1,10 @@ (file-sets)= # Working with local files - + To build a local project in a Nix derivation, its source files must be [accessible](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) @@ -94,11 +98,11 @@ and set up `niv` to pin the Nixpkgs dependency: ```shell-session $ mkdir fileset $ cd fileset -$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch nixos-unstable" +$ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch master" ``` :::{note} -We're using the `nixos-unstable` channel branch here, because the library is part of NixOS 23.11 and up, which has not been released at the time of writing. +We're using the `master` branch here, because the covered library features are part of NixOS 23.11 and up, which has not been released at the time of writing. ::: Then create a `default.nix` file with the following contents: @@ -718,4 +722,4 @@ trace: - world.txt (regular) We have shown some examples on how to use all of the fundamental file set functions. For more complex use cases, they can be composed as needed. -For the complete list and more details, see the [`lib.fileset` reference documentation](https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset). +For the complete list and more details, see the [`lib.fileset` reference documentation](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset). From 69e6dd6d9d3aed8c6cd64a9aa094bd9b7eab9fb8 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 21:32:14 +0100 Subject: [PATCH 47/50] Some changes from a full read --- source/tutorials/file-sets.md | 64 +++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 30 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 5d39227f2d..bfe2b57064 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -7,9 +7,10 @@ instead of https://nixos.org/manual/nixpkgs/unstable/#sec-fileset --> To build a local project in a Nix derivation, -its source files must be [accessible](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) +its source files must be accessible to the [derivation builder](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). -Since by default the `builder` runs in an isolated environment that only allows reading from the Nix store, +Since by default, derivations get built in an [isolated environment](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) +that only allows reading from the Nix store, the Nix language has built-in features to copy local files to the store and expose the resulting store paths. Using these features directly can be tricky however: @@ -23,23 +24,27 @@ Using these features directly can be tricky however: can address these problems. However, it's often hard to express the desired path selection using the `filter` function interface. -In this tutorial you'll learn how to use the Nixpkgs [`fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. -It abstracts over built-in functionality and offers a safer and more convenient interface. +In this tutorial you'll learn how to use Nixpkgs' [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. +It abstracts over built-in functionality with a safer and more convenient interface. ## File sets -The file set library is based on the concept of _file sets_, +The basic concept is that of a _file set_, a data type representing a collection of local files. File sets can be created, composed, and manipulated with the various functions of the library. You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): ```shell-session -$ nix repl -f channel:nixos-unstable +$ nix repl -f https://github.com/NixOS/nixpkgs/tarball/master ... nix-repl> fs = lib.fileset ``` +:::{note} +We're using the `master` branch here, because the covered library features are part of NixOS 23.11 and up, which has not been released at the time of writing. +::: + The [`trace`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.trace) function pretty-prints the files included in a given file set: ```shell-session @@ -48,7 +53,7 @@ trace: /home/user (all files in directory) null ``` -All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path). +All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path) instead. Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path. In the previous trace this is indicated by `(all files in directory)`. @@ -64,14 +69,14 @@ trace: /home/user (all files in directory) ::: Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested. -You don't have to worry as much about accidentally copying secrets into the world-readable store. +So You don't have to worry as much about accidentally copying secrets into the world-readable store. In this example, although we pretty-printed the home directory, no files were copied. This is in contrast to coercion of paths to strings such as in `"${./.}"`, which copies the whole directory to the Nix store on evaluation! :::{warning} -With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), a local directory containing `flake.nix` is always copied into the Nix store *completely* unless it is a Git repository! +When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), local directories with a `flake.nix` are always copied into the Nix store *completely* unless it is a Git repository! ::: This implicit coercion also works for files: @@ -101,9 +106,6 @@ $ cd fileset $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch master" ``` -:::{note} -We're using the `master` branch here, because the covered library features are part of NixOS 23.11 and up, which has not been released at the time of writing. -::: Then create a `default.nix` file with the following contents: @@ -130,13 +132,14 @@ $ echo hello > hello.txt $ echo world > world.txt ``` -Start with a minimal `build.nix` based on `stdenv.mkDerivation`: +Start with a minimal `build.nix` based on [`stdenv.mkDerivation`](https://nixos.org/manual/nixpkgs/stable/#part-stdenv): ```{code-block} nix :caption: build.nix { stdenv }: stdenv.mkDerivation { name = "fileset"; + # This doesn't use the file set library just yet! src = ./.; postInstall = '' mkdir $out @@ -158,15 +161,16 @@ building '/nix/store/36p7xhc0rr8jvslban0zba0f7aij8cmb-fileset.drv'... /nix/store/zdljz7v5bhv1nnh5mdh3xf418cf7z622-fileset ``` -You may have already noticed a problem: -Only one file is needed to build the derivation. -But passing the path to `hello.txt` will result in an error, because `mkDerivation` expects a directory: +Since only `hello.txt` is needed to build the derivation, +we can try passing it directly in `src`. +But this will result in an error, because `mkDerivation` expects a directory: ```{code-block} diff :caption: build.nix { stdenv }: stdenv.mkDerivation { name = "fileset"; + # This doesn't use the file set library just yet! - src = ./.; + src = ./hello.txt; postInstall = '' @@ -312,7 +316,7 @@ result -> /nix/store/xknflcvjaa8dj6a6vkg629zmcrgz10rh-fileset Since `src` refers to the whole directory, and its contents change when `nix-build` succeeds, Nix will have to start over every time. :::{note} -This will also happen without the file set library, e.g. when setting `src = ./.;` directly. +This will also happen with `src = ./.`, without using the file set library. ::: The [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function subtracts one file set from another. @@ -370,7 +374,7 @@ Removing the `./result` symlink creates a new problem, though: $ rm result $ nix-build error: lib.fileset.difference: Second argument (negative set) - (/home/user/select/result) is a path that does not exist. + (/home/user/fileset/result) is a path that does not exist. To create a file set from a path that may not exist, use `lib.fileset.maybeMissing`. ``` @@ -392,7 +396,7 @@ This now works, using the whole directory since `./result` is not present: ``` $ nix-build -trace: /home/user/select (all files in directory) +trace: /home/user/fileset (all files in directory) this derivation will be built: /nix/store/zr19bv51085zz005yk7pw4s9sglmafvn-fileset.drv ... @@ -427,7 +431,7 @@ Again, Nix will start from scratch: ```shell-session $ nix-build -trace: /home/user/select +trace: /home/user/fileset trace: - default.nix (regular) trace: - nix (all files in directory) trace: - build.nix (regular) @@ -460,9 +464,7 @@ Create a file set containing a union of the files to exclude (`fs.unions [ ... ] in ``` -Changing any of the excluded files now doesn't necessarily cause a new build anymore. - -Check it and modify one of the excluded files again: +Changing any of the excluded files now doesn't necessarily cause a new build anymore: ``` $ echo >> build.nix @@ -597,7 +599,7 @@ this derivation will be built: Only the specified files are used, even when a new one is added: ```shell-session -$ touch src/select.o src/README.md +$ touch src/select.o README.md $ nix-build trace: - build.sh (regular) @@ -611,13 +613,13 @@ trace: - world.txt (regular) ## Matching files tracked by Git -If a directory is part of a Git repository, [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) automatically filters for files that are tracked by Git. +If a directory is part of a Git repository, passing it to [`gitTracked`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) gives you a file set that only includes files tracked by Git. Create a local Git repository and add all files except `src/select.o` and `./result` to it: ```shell-session $ git init -Initialized empty Git repository in /home/user/select/.git/ +Initialized empty Git repository in /home/user/fileset/.git/ $ git add -A $ git reset src/select.o result ``` @@ -649,8 +651,9 @@ Building we get ```shell-session $ nix-build -warning: Git tree '/home/user/select' is dirty +warning: Git tree '/home/user/fileset' is dirty trace: /home/vg/src/nix.dev/fileset +trace: - README.md (regular) trace: - build.nix (regular) trace: - build.sh (regular) trace: - default.nix (regular) @@ -669,7 +672,8 @@ this derivation will be built: This includes too much though, as not all of these files are needed to build the derivation as originally intended. :::{note} -With [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with +When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with + ```shell-session $ nix build path:. @@ -681,7 +685,7 @@ However it's also not needed, because by default, `nix build` only allows access ## Intersection This is where `intersection` comes in. -It allows creating a file set that consists only of files that are in _both_ of the two given file sets. +It allows creating a file set that consists only of files that are in _both_ of two given file sets. Select all files that are both tracked by Git *and* relevant for the build: @@ -707,7 +711,7 @@ This will produce the same output as in the other approach and therefore re-use ```shell-session $ nix-build -warning: Git tree '/home/user/select' is dirty +warning: Git tree '/home/user/fileset' is dirty trace: - build.sh (regular) trace: - hello.txt (regular) trace: - src From 9ecfd01a0d7431f603ce5e11b9568ed3dd52cc84 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Wed, 22 Nov 2023 21:47:35 +0100 Subject: [PATCH 48/50] Remove unecessary non-fileset part --- source/tutorials/file-sets.md | 78 +++-------------------------------- 1 file changed, 5 insertions(+), 73 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index bfe2b57064..9d43455e1b 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -106,7 +106,6 @@ $ cd fileset $ nix-shell -p niv --run "niv init --nixpkgs nixos/nixpkgs --nixpkgs-branch master" ``` - Then create a `default.nix` file with the following contents: ```{code-block} nix @@ -132,79 +131,12 @@ $ echo hello > hello.txt $ echo world > world.txt ``` -Start with a minimal `build.nix` based on [`stdenv.mkDerivation`](https://nixos.org/manual/nixpkgs/stable/#part-stdenv): - -```{code-block} nix -:caption: build.nix -{ stdenv }: -stdenv.mkDerivation { - name = "fileset"; - # This doesn't use the file set library just yet! - src = ./.; - postInstall = '' - mkdir $out - cp -v hello.txt $out - ''; -} -``` - -Try it out (note that it will take some time to fetch the first time around): - -```shell-session -$ nix-build -this derivation will be built: - /nix/store/36p7xhc0rr8jvslban0zba0f7aij8cmb-fileset.drv -building '/nix/store/36p7xhc0rr8jvslban0zba0f7aij8cmb-fileset.drv'... -... -'hello.txt' -> '/nix/store/zdljz7v5bhv1nnh5mdh3xf418cf7z622-fileset/hello.txt' -... -/nix/store/zdljz7v5bhv1nnh5mdh3xf418cf7z622-fileset -``` - -Since only `hello.txt` is needed to build the derivation, -we can try passing it directly in `src`. -But this will result in an error, because `mkDerivation` expects a directory: - -```{code-block} diff -:caption: build.nix - { stdenv }: - stdenv.mkDerivation { - name = "fileset"; - # This doesn't use the file set library just yet! -- src = ./.; -+ src = ./hello.txt; - postInstall = '' - mkdir $out - cp -v hello.txt $out - ''; - } -``` - -```shell-session -$ nix-build -this derivation will be built: - /nix/store/kaik07l2i7s7wh7lcxpwynv4lmn4h742-fileset.drv -building '/nix/store/kaik07l2i7s7wh7lcxpwynv4lmn4h742-fileset.drv'... -unpacking sources -unpacking source archive /nix/store/i9pmrzmpshapij2kin22pff6fc2adavx-hello.txt -do not know how to unpack source archive /nix/store/i9pmrzmpshapij2kin22pff6fc2adavx- -hello.txt -error: builder for '/nix/store/kaik07l2i7s7wh7lcxpwynv4lmn4h742-fileset.drv' failed w -ith exit code 1; - last 3 log lines: - > unpacking sources - > unpacking source archive /nix/store/i9pmrzmpshapij2kin22pff6fc2adavx-hello.t -xt - > do not know how to unpack source archive /nix/store/i9pmrzmpshapij2kin22pff6 -fc2adavx-hello.txt - For full logs, run 'nix log /nix/store/kaik07l2i7s7wh7lcxpwynv4lmn4h742-filese -t.drv'. -``` - ## Adding files to the Nix store -That's where [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource) comes in: -It adds exactly the files in the given file set to a directory in the Nix store, starting from a specified root path. +To add files in a given file set to the Nix store, +we use [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource). +This function needs a `root` attribute to know which path to use as the root of the resulting store path, +but only exactly the files in the `fileset` argument are included in the result. Define `build.nix` as follows: @@ -233,7 +165,7 @@ stdenv.mkDerivation { The call to `fs.trace` prints the file set that will be used as a derivation input. -The build will succeed: +The build will succeed (note that it will take some time to fetch the first time around): ``` $ nix-build From fb3fb0c06068ef0531db72c1d30b5235fa9911a1 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Thu, 23 Nov 2023 15:58:44 +0100 Subject: [PATCH 49/50] Update source/tutorials/file-sets.md Co-authored-by: Valentin Gagarin --- source/tutorials/file-sets.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 9d43455e1b..93495d7372 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -6,12 +6,8 @@ Also use https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-filese instead of https://nixos.org/manual/nixpkgs/unstable/#sec-fileset --> -To build a local project in a Nix derivation, -its source files must be accessible -to the [derivation builder](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). -Since by default, derivations get built in an [isolated environment](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) -that only allows reading from the Nix store, -the Nix language has built-in features to copy local files to the store and expose the resulting store paths. +To build a local project in a Nix derivation, source files must be accessible to its [`builder` executable](https://nixos.org/manual/nix/stable/language/derivations#attr-builder). +Since by default, the `builder` runs in an [isolated environment](https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-sandbox) that only allows reading from the Nix store, the Nix language has built-in features to copy local files to the store and expose the resulting store paths. Using these features directly can be tricky however: From b6700a9c45e08766d500b9ac5f6a4045fce5be19 Mon Sep 17 00:00:00 2001 From: Silvan Mosberger Date: Thu, 23 Nov 2023 17:05:37 +0100 Subject: [PATCH 50/50] Review in meeting --- source/tutorials/file-sets.md | 179 ++++++++++++++-------------------- 1 file changed, 71 insertions(+), 108 deletions(-) diff --git a/source/tutorials/file-sets.md b/source/tutorials/file-sets.md index 93495d7372..4b8934c723 100644 --- a/source/tutorials/file-sets.md +++ b/source/tutorials/file-sets.md @@ -20,13 +20,12 @@ Using these features directly can be tricky however: can address these problems. However, it's often hard to express the desired path selection using the `filter` function interface. -In this tutorial you'll learn how to use Nixpkgs' [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. -It abstracts over built-in functionality with a safer and more convenient interface. +In this tutorial you'll learn how to use the Nixpkgs [`lib.fileset` library](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset) to work with local files in derivations. +It abstracts over built-in functionality and offers a safer and more convenient interface. ## File sets -The basic concept is that of a _file set_, -a data type representing a collection of local files. +A _file set_ is a data type representing a collection of local files. File sets can be created, composed, and manipulated with the various functions of the library. You can explore and learn about the library with [`nix repl`](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-repl): @@ -49,7 +48,7 @@ trace: /home/user (all files in directory) null ``` -All functions that expect a file set for an argument also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path) instead. +All functions that expect a file set for an argument can also accept a [path](https://nixos.org/manual/nix/stable/language/values#type-path). Such path arguments are then [implicitly turned into sets](https://nixos.org/manual/nixpkgs/unstable/#sec-fileset-path-coercion) that contain _all_ files under the given path. In the previous trace this is indicated by `(all files in directory)`. @@ -65,14 +64,15 @@ trace: /home/user (all files in directory) ::: Even though file sets conceptually contain local files, these files are *never* added to the Nix store unless explicitly requested. -So You don't have to worry as much about accidentally copying secrets into the world-readable store. +Therefore you don't have to worry as much about accidentally copying secrets into the world-readable store. In this example, although we pretty-printed the home directory, no files were copied. This is in contrast to coercion of paths to strings such as in `"${./.}"`, which copies the whole directory to the Nix store on evaluation! :::{warning} -When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), local directories with a `flake.nix` are always copied into the Nix store *completely* unless it is a Git repository! +When using the [`flakes` and `nix-command` experimental features](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), +a local directory within a Flake is always copied into the Nix store *completely* unless it is a Git repository! ::: This implicit coercion also works for files: @@ -129,10 +129,9 @@ $ echo world > world.txt ## Adding files to the Nix store -To add files in a given file set to the Nix store, -we use [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource). -This function needs a `root` attribute to know which path to use as the root of the resulting store path, -but only exactly the files in the `fileset` argument are included in the result. +Files in a given file set can be added to the Nix store with [`toSource`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.toSource). +The argument to this function requires a `root` attribute to determine which source directory to copy to the store. +Only the files in the `fileset` attribute are included in the result. Define `build.nix` as follows: @@ -161,7 +160,11 @@ stdenv.mkDerivation { The call to `fs.trace` prints the file set that will be used as a derivation input. -The build will succeed (note that it will take some time to fetch the first time around): +Try building it: + +:::{note} +It will take a while to fetch Nixpkgs the first time around. +::: ``` $ nix-build @@ -244,7 +247,7 @@ result -> /nix/store/xknflcvjaa8dj6a6vkg629zmcrgz10rh-fileset Since `src` refers to the whole directory, and its contents change when `nix-build` succeeds, Nix will have to start over every time. :::{note} -This will also happen with `src = ./.`, without using the file set library. +This will also happen without the file set library, e.g. when setting `src = ./.;` directly. ::: The [`difference`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.difference) function subtracts one file set from another. @@ -374,22 +377,17 @@ One way to fix this is to use [`unions`](https://nixos.org/manual/nixpkgs/unstab Create a file set containing a union of the files to exclude (`fs.unions [ ... ]`), and subtract it (`difference`) from the complete directory (`./.`): -```{code-block} diff +```{code-block} nix :caption: build.nix - { stdenv, lib }: - let - fs = lib.fileset; -- sourceFiles = fs.difference ./. (fs.maybeMissing ./result); -+ sourceFiles = -+ fs.difference -+ ./. -+ (fs.unions [ -+ (fs.maybeMissing ./result) -+ ./default.nix -+ ./build.nix -+ ./nix -+ ]); - in + sourceFiles = + fs.difference + ./. + (fs.unions [ + (fs.maybeMissing ./result) + ./default.nix + ./build.nix + ./nix + ]); ``` Changing any of the excluded files now doesn't necessarily cause a new build anymore: @@ -408,16 +406,12 @@ trace: - world.txt (regular) ## Filter -With the [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function, -you can find sets of files satisfying a criteria. +The [`fileFilter`](https://nixos.org/manual/nixpkgs/unstable/#function-library-lib.fileset.fileFilter) function allows filtering file sets such that each included file satisfies the given criteria. Use it to select all files with a name ending in `.nix`: ```{code-block} diff :caption: build.nix - { stdenv, lib }: - let - fs = lib.fileset; sourceFiles = fs.difference ./. @@ -428,7 +422,6 @@ Use it to select all files with a name ending in `.nix`: + (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) ./nix ]); - in ``` This does not change the result, even if we add a new `.nix` file. @@ -458,47 +451,37 @@ $ touch build.sh src/select.{c,h} Then create a file set from only the files to be included explicitly: -```{code-block} diff +```{code-block} nix :caption: build.nix - { stdenv, lib }: - let - fs = lib.fileset; -- sourceFiles = -- fs.difference -- ./. -- (fs.unions [ -- (fs.maybeMissing ./result) -- (fs.fileFilter (file: lib.hasSuffix ".nix" file.name) ./.) -- ./nix -- ]); -+ sourceFiles = fs.unions [ -+ ./hello.txt -+ ./world.txt -+ ./build.sh -+ (fs.fileFilter -+ (file: -+ lib.hasSuffix ".c" file.name -+ || lib.hasSuffix ".h" file.name -+ ) -+ ./src -+ ) -+ ]; - in +{ stdenv, lib }: +let + fs = lib.fileset; + sourceFiles = fs.unions [ + ./hello.txt + ./world.txt + ./build.sh + (fs.fileFilter + (file: + lib.hasSuffix ".c" file.name + || lib.hasSuffix ".h" file.name + ) + ./src + ) + ]; +in - fs.trace sourceFiles +fs.trace sourceFiles - stdenv.mkDerivation { - name = "fileset"; - src = fs.toSource { - root = ./.; - fileset = sourceFiles; - }; - postInstall = '' -- mkdir $out -- cp -v {hello,world}.txt $out -+ cp -vr . $out - ''; - } +stdenv.mkDerivation { + name = "fileset"; + src = fs.toSource { + root = ./.; + fileset = sourceFiles; + }; + postInstall = '' + cp -vr . $out + ''; +} ``` The `postInstall` script is simplified to rely on the sources to be pre-filtered appropriately: @@ -552,30 +535,14 @@ $ git add -A $ git reset src/select.o result ``` -Now we can re-use this selection of files using `gitTracked`: +Re-use this selection of files with `gitTracked`: -```{code-block} diff +```{code-block} nix :caption: build.nix - { stdenv, lib }: - let - fs = lib.fileset; -- sourceFiles = fs.unions [ -- ./hello.txt -- ./world.txt -- ./build.sh -- (fs.fileFilter -- (file: -- lib.hasSuffix ".c" file.name -- || lib.hasSuffix ".h" file.name -- ) -- ./src -- ) -- ]; -+ sourceFiles = fs.gitTracked ./.; - in + sourceFiles = fs.gitTracked ./.; ``` -Building we get +Build it again: ```shell-session $ nix-build @@ -600,7 +567,8 @@ this derivation will be built: This includes too much though, as not all of these files are needed to build the derivation as originally intended. :::{note} -When using [experimental Flakes](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with +When using the [`flakes` and `nix-command` experimental features](https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-flake), +it's [not really possible](https://github.com/NixOS/nix/issues/9292) to use this function, even with ```shell-session @@ -617,22 +585,17 @@ It allows creating a file set that consists only of files that are in _both_ of Select all files that are both tracked by Git *and* relevant for the build: -```{code-block} diff +```{code-block} nix :caption: build.nix - { stdenv, lib }: - let - fs = lib.fileset; -- sourceFiles = fs.gitTracked ./.; -+ sourceFiles = -+ fs.intersection -+ (fs.gitTracked ./.) -+ (fs.unions [ -+ ./hello.txt -+ ./world.txt -+ ./build.sh -+ ./src -+ ]); - in + sourceFiles = + fs.intersection + (fs.gitTracked ./.) + (fs.unions [ + ./hello.txt + ./world.txt + ./build.sh + ./src + ]); ``` This will produce the same output as in the other approach and therefore re-use a previous build result: