nixos/etc-overlay: ship empty /etc/machine-id placeholder on immutable /etc

[r-vdp]

#523894 made /etc/machine-id a symlink to /var/lib/nixos/machine-id and
created the target with content uninitialized.
Unfortunately the generated id is never committed, because:

• systemd-machine-id-commit.service has ConditionPathIsReadWrite=/etc,
  which is always false on the read-only overlay, so the unit is skipped.
• Even with that condition cleared, machine_id_commit() checks whether
  /etc/machine-id itself is a mountpoint (no symlink following), finds the
  symlink isn't, and exits without writing.

So /var/lib/nixos/machine-id stays uninitialized forever, every boot is
ConditionFirstBoot=yes, and the machine-id is random on every boot.

I think it's more correct to have an empty regular file in the etc image so that:

• systemd bind-mounts /run/machine-id over it for the session.
• ConditionFirstBoot is no, which is the only logical default for a file
  that can never change. The previous behaviour fired first-boot units on every boot.
• systemd-machine-id-commit is cleanly condition-skipped, no failed unit.
• Users who want persistence can bind a writable backing file onto the file
  and have commit write through it, which does not work with a symlink.

The whole attrset is mkDefault, so any user definition of
environment.etc.machine-id overrides it.

For users who need a persistent ID, there are two main options:

• bind-mount a writable backing file from initrd (and clear the commit condition)
• pass systemd.machine_id= / systemd.machine_id=firmware on the kernel command line.

cc @anglesideangle @rebmit @nikstur

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.
• Follows the automation/AI policy.

[rebmit]

the generated id is never committed

You're right, I missed that in the previous PR. The symlink setup requires modifying systemd-machine-id-commit.service like this to work, which guarantees proper first-boot semantics as long as /var/lib/nixos is persisted.

  environment.etc."machine-id" = {
    source = "/var/lib/nixos/etc/machine-id";
    mode = "direct-symlink";
  };

  boot.initrd.systemd.tmpfiles.settings.machine-id."/sysroot/var/lib/nixos/etc/machine-id".f = {
    argument = "uninitialized";
  };

  systemd.services.systemd-machine-id-commit = {
    unitConfig.ConditionPathIsMountPoint = [
      ""
      "/var/lib/nixos/etc/machine-id"
    ];
    serviceConfig.ExecStart = [
      ""
      "systemd-machine-id-setup --commit --root /var/lib/nixos"
    ];
  };

I think it's more correct to have an empty regular file in the etc image

That makes sense. Though personally, I still slightly prefer the symlink approach for consistency with how we handle userborn / sysusers.

[r-vdp]

Yeah, I would prefer to make minimal changes to upstream services though, and stay closer to what upstream expects, to lower the maintenance burden.

[nikstur]

@WilliButz might be able to give more context on how this used to work with symlinks and why now it doesn't.

[WilliButz]

There is a bunch of context for this in this upstream issue: systemd/systemd#39438

The underlying change mentioned there breaks a couple of setups that do not seem to be supported/intended by upstream (which I gathered from discussions in person after the issue stalled), i.e. any setup that tries to persist only the machine-id outside /etc via symlink or bind-mount. Breaking in the sense that symlinks are no longer followed (since 258) and a bind-mounted machine-id file doesn't work without changes to upstream units. Same for systemd-firstboo, which also no longer works in such a setup without changes to upstream units. (as mentioned in comments above)

The two explicitly supported options seem to be limited to persisting all of /etc, including a generated machine-id, or not persisting a generated machine-id at all.

Anything that does not fit with how upstream expects a system to be set up could break just like the symlink setup did. So I think the correct thing to do would be to find some way to align our and upstream's expectations on this. The linked issue above was an attempt to communicate an actual use case that is not "just" a niche NixOS thing, but could be valuable to other platforms/setups as well (as shown by others that commented there) and then for example simply going back to resolving symlinks again, but with explicit support. Unfortunately the discussion on the issue stalled, but I don't think it's entirely hopeless.

[r-vdp]

I guess the question remains though, what do we do right now? Because what we currently have on master is not working.

Do we ship this PR as a solution for now that's at least workable, until upstream comes with a better fix? Or does anyone have another suggestion?

[rebmit]

Do we ship this PR as a solution for now that's at least workable, until upstream comes with a better fix?

I agree.

[anglesideangle]

Sorry for not catching this in the original PR. It's unfortunate that systemd does not consider symlinked /etc/machine-id to be supported.

For users who need a persistent ID, there are two main options:

• bind-mount a writable backing file from initrd (and clear the commit condition)
• pass systemd.machine_id= / systemd.machine_id=firmware on the kernel command line.

I think it would be useful to document this in the etc-overlay section of the nixos manual