Boot stuck due to AMDGPU memory leak after NixOS bump

[msanft]

After bumping my system from e99366c to 85a6c4a, my system got stuck in my motherboard's boot splash screen after unlocking the disk.

No firmware updates took place, so the cause is likely in userspace. After re-booting into an older generation, everything worked again. I examined the dmesg output and was presented with floods of the following:

Nov 17 09:03:09 bastion kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Not enough memory for command submission!
Nov 17 09:03:09 bastion kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Not enough memory for command submission!
Nov 17 09:03:09 bastion kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Not enough memory for command submission!
Nov 17 09:03:09 bastion kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Not enough memory for command submission!
Nov 17 09:03:09 bastion kernel: [drm:amdgpu_cs_ioctl [amdgpu]] *ERROR* Not enough memory for command submission!

Reading about this for a little, it seems that this error is typically caused by userspace programs leaking graphics memory.

This issue occurred on a setup with a AMD Ryzen 9 9950X CPU and a NVIDIA RTX 3070 GPU, running the latest kernel (i.e., linuxPackages_latest).

[msanft]

Without setting boot.kernelPackages at all (i.e. with the 6.12 kernel), the issues doesn't occur.

EDIT: After a while, I randomly ran into this crash with the 6.12 kernel:

Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:7 pasid:32773)
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:  in process chrome pid 4581 thread chrome:cs0 pid 4613
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:   in page starting at address 0x000000003d800000 from client 0x1b (UTCL2)
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00701430
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:          Faulty UTCL2 client ID: SQC (data) (0xa)
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:          MORE_FAULTS: 0x0
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:          WALKER_ERROR: 0x0
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:          PERMISSION_FAULTS: 0x3
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:          MAPPING_ERROR: 0x0
Nov 17 10:38:59 bastion kernel: amdgpu 0000:6b:00.0: amdgpu:          RW: 0x0
Nov 17 10:39:09 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: Dumping IP State
Nov 17 10:39:09 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: Dumping IP State Completed
Nov 17 10:39:09 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: ring gfx_0.0.0 timeout, but soft recovered
Nov 17 10:39:09 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: Dumping IP State
Nov 17 10:39:09 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: Dumping IP State Completed
Nov 17 10:39:09 bastion kernel: amdgpu 0000:6b:00.0: amdgpu: ring gfx_0.1.0 timeout, but soft recovered

[msanft]

Downgrading to older kernel versions (6.17.{5,6,7}) didn't help either.

[msanft]

Diff between the 2 generations:

[msanft@bastion:/nix/var/nix/profiles]$ nvd diff system-31-link system-32-link
<<< system-31-link
>>> system-32-link
Version changes:
[U*]  #001  1password                      8.11.16 -> 8.11.18
[U.]  #002  adwaita-fonts                  48.2 -> 49.0
[U*]  #003  adwaita-icon-theme             48.0 -> 49.0
[U*]  #004  at-spi2-core                   2.56.2 -> 2.58.0
[U*]  #005  atuin                          18.8.0 -> 18.10.0
[U.]  #006  bash-completion                2.16.0 -> 2.17.0
[U.]  #007  btrfs-progs                    6.17 -> 6.17.1
[C.]  #008  c-ares                         1.34.5 -> 1.34.5 x3
[U.]  #009  dash                           0.5.12 -> 0.5.13.1
[U*]  #010  decibels                       48.0 -> 49.0
[C.]  #011  electron                       37.9.0, 38.6.0 -> 37.10.0, 38.7.0
[C.]  #012  electron-unwrapped             37.9.0, 38.6.0 -> 37.10.0, 38.7.0
[U.]  #013  elfutils                       0.193 x2 -> 0.194 x2
[U.]  #014  evolution-data-server          3.56.2 x2 -> 3.58.1 x2
[U.]  #015  ffmpeg                         7.1.1-data, 7.1.1-lib -> 7.1.2-data, 7.1.2-lib, 8.0-data, 8.0-lib
[U.]  #016  ffmpeg-headless                7.1.1-data x2, 7.1.1-lib x2 -> 8.0-data x2, 8.0-lib x2
[U*]  #017  firefox                        144.0.2 -> 145.0
[U.]  #018  firefox-unwrapped              144.0.2 -> 145.0
[U.]  #019  fluidsynth                     2.4.8 -> 2.5.0
[U.]  #020  gdk-pixbuf                     2.42.12 -> 2.44.3
[C*]  #021  gdm                            <none>, 48.0 -> <none>, 49.1
[U*]  #022  gh                             2.83.0 -> 2.83.1
[U*]  #023  git                            2.51.0, 2.51.0-doc -> 2.51.2, 2.51.2-doc
[U.]  #024  git-minimal                    2.51.0 -> 2.51.2
[U.]  #025  gjs                            1.84.2 -> 1.86.0
[U*]  #026  glib                           2.84.4 x2, 2.84.4-bin, 2.84.4-dev -> 2.86.1 x2, 2.86.1-bin, 2.86.1-dev
[C.]  #027  glibmm                         2.66.8 x2, 2.84.0 -> 2.66.8 x2, 2.86.0
[U.]  #028  glycin-loaders                 1.2.3 -> 2.0.5
[U.]  #029  gnome-clocks                   48.0 -> 49.0
[U*]  #030  gnome-control-center           48.4 -> 49.1
[U.]  #031  gnome-desktop                  44.3 -> 44.4
[U*]  #032  gnome-font-viewer              48.0 -> 49.0
[C*]  #033  gnome-initial-setup            <none>, 48.1 -> <none>, 49.0
[U*]  #034  gnome-online-accounts          3.54.5, 3.54.5-man -> 3.56.1, 3.56.1-man
[U*]  #035  gnome-remote-desktop           48.1 -> 49.1
[U*]  #036  gnome-session                  48.0, 48.0-sessions -> 49.1, 49.1-sessions
[U.]  #037  gnome-session-ctl              47.0.1 -> 49.0
[U*]  #038  gnome-settings-daemon          48.1, 48.1-gsettings-schemas -> 49.1, 49.1-gsettings-schemas
[U*]  #039  gnome-shell                    48.4 -> 49.1
[U*]  #040  gnome-system-monitor           48.1 -> 49.1
[U*]  #041  go                             1.25.2 -> 1.25.3
[U.]  #042  gobject-introspection          1.84.0 -> 1.86.0, 1.86.0-dev
[U.]  #043  gobject-introspection-wrapped  1.84.0 -> 1.86.0, 1.86.0-dev
[U.]  #044  gsettings-desktop-schemas      48.0 -> 49.0
[U*]  #045  gtk+3                          3.24.49 -> 3.24.51
[U.]  #046  gtk4                           4.18.6 -> 4.20.2
[U.]  #047  gtkmm                          4.18.0 -> 4.20.0
[C.]  #048  gtksourceview                  4.8.4 -> 4.8.4, 5.18.0
[U*]  #049  gvfs                           1.57.2 x2 -> 1.58.0 x2
[C.]  #050  icu4c                          75.1, 76.1 x2, 76.1-dev, 77.1 x2 -> 76.1 x2, 76.1-dev, 77.1 x2
[U.]  #051  inih                           61 -> 62
[U.]  #052  initrd-linux                   6.17.7 -> 6.17.8
[U*]  #053  k6                             1.3.0 -> 1.4.0
[U*]  #054  kexec-tools                    2.0.31 x2 -> 2.0.32 x2
[U*]  #055  kubernetes-helm                3.19.0 -> 3.19.1
[C.]  #056  lcms2                          2.17 x2 -> 2.17 x2, 2.17-bin, 2.17-dev
[U.]  #057  libadwaita                     1.7.6 -> 1.8.0
[C.]  #058  libarchive                     3.8.0-lib, 3.8.1-lib x2 -> 3.8.0-lib, 3.8.2-lib x2
[U.]  #059  libdecor                       0.2.3 -> 0.2.4
[U.]  #060  libdrm                         2.4.126 x2, 2.4.126-bin, 2.4.126-dev -> 2.4.127 x2, 2.4.127-bin, 2.4.127-dev
[U.]  #061  libedit                        20250104-3.1 -> 20251016-3.1
[U.]  #062  libinput                       1.29.1 -> 1.29.2
[U.]  #063  libpq                          18.0 -> 18.1
[U.]  #064  librsvg                        2.61.1 -> 2.61.2
[C.]  #065  libseccomp                     2.6.0-lib x3 -> 2.6.0, 2.6.0-dev, 2.6.0-lib x3
[U.]  #066  libtraceevent                  1.8.4 -> 1.8.5
[C.]  #067  libunistring                   1.3, 1.4 x2 -> 1.3, 1.4.1 x2
[C.]  #068  libxml2                        2.13.8, 2.15.0 x2 -> 2.13.8, 2.15.1 x2
[U.]  #069  linux                          6.17.7, 6.17.7-modules x2, 6.17.7-modules-shrunk -> 6.17.8, 6.17.8-modules x2, 6.17.8-modules-shrunk
[U.]  #070  linux-firmware                 20251021-zstd -> 20251111-zstd
[U*]  #071  localsearch                    3.9.0 -> 3.10.1
[U*]  #072  loupe                          48.1 -> 49.1
[U.]  #073  mesa                           25.2.6 -> 25.3.0
[U.]  #074  mutter                         48.4 -> 49.1.1
[U*]  #075  nautilus                       48.3 -> 49.1
[U*]  #076  networkmanager                 1.52.1, 1.52.1-doc, 1.52.1-man -> 1.54.1, 1.54.1-doc, 1.54.1-man
[U.]  #077  nixos-system-bastion           25.11.20251111.9da7f1c -> 25.11.20251116.85a6c4a
[U*]  #078  nodejs                         22.20.0 -> 22.21.1
[U.]  #079  nvidia-container-toolkit       1.17.9, 1.17.9-tools -> 1.18.0, 1.18.0-tools
[U.]  #080  nvidia-open                    6.17.7-580.95.05 -> 6.17.8-580.105.08
[U*]  #081  nvidia-settings                580.95.05 -> 580.105.08
[U*]  #082  nvidia-x11                     580.95.05-6.17.7, 580.95.05-6.17.7-bin, 580.95.05-6.17.7-firmware -> 580.105.08-6.17.8, 580.105.08-6.17.8-bin, 580.105.08-6.17.8-firmware
[U*]  #083  obsidian                       1.9.14 -> 1.10.3
[U.]  #084  onetbb                         2022.2.0 -> 2022.3.0
[U.]  #085  openjpeg                       2.5.2 x2 -> 2.5.4 x2
[U*]  #086  openssh                        10.1p1 -> 10.2p1, 10.2p1-man
[U.]  #087  p11-kit                        0.25.9 x2, 0.25.9-bin -> 0.25.10 x2, 0.25.10-bin
[U.]  #088  pango                          1.56.4 x2, 1.56.4-bin -> 1.57.0 x2, 1.57.0-bin
[U.]  #089  protobuf                       32.1 -> 33.0
[C.]  #090  publicsuffix-list              0-unstable-2025-03-12, 0-unstable-2025-08-28 x2 -> 0-unstable-2025-03-12, 0-unstable-2025-10-08 x2
[C*]  #091  python3                        3.12.11, 3.12.11-env, 3.13.8 x2, 3.13.8-env x4 -> 3.12.12, 3.12.12-env, 3.13.9 x2, 3.13.9-env x5
[U.]  #092  python3.12-cryptography        46.0.1 -> 46.0.2
[U.]  #093  python3.12-grpcio              1.75.1 -> 1.76.0
[U.]  #094  python3.12-numpy               2.3.3 -> 2.3.4
[U.]  #095  python3.12-protobuf            6.32.1 -> 6.33.0
[U*]  #096  python3.13                     3.13.8-doc -> 3.13.9-doc
[U.]  #097  python3.13-pygobject           3.50.0 -> 3.54.3, 3.54.3-dev
[U*]  #098  rygel                          0.44.2 -> 45.0
[C.]  #099  s2n-tls                        1.5.17, 1.5.26 -> 1.5.17, 1.5.27
[U.]  #100  sdl2-compat                    2.32.56 -> 2.32.58
[U.]  #101  sdl3                           3.2.24-lib -> 3.2.26-lib
[U.]  #102  spidermonkey                   128.14.0 -> 140.4.0
[C*]  #103  systemd                        <none>, 258 x2, 258-man -> <none>, 258.1 x2, 258.1-man
[U.]  #104  systemd-minimal                258 x2 -> 258.1 x2
[U.]  #105  systemd-minimal-libs           258 x2 -> 258.1 x2
[U*]  #106  tailscale                      1.88.4 -> 1.90.6
[U.]  #107  tecla                          48.0.2 -> 49.0
[U*]  #108  thunderbird                    144.0.1 -> 145.0
[U.]  #109  thunderbird-unwrapped          144.0.1 -> 145.0
[U*]  #110  tinysparql                     3.9.2 -> 3.10.1
[U.]  #111  unbound                        1.24.0-lib x2 -> 1.24.1-lib x2
[C.]  #112  unit-plymouth-quit.service     <none> -> <none> x2
[U*]  #113  uv                             0.8.23 -> 0.9.7
[U.]  #114  v4l-utils                      1.30.1 x2 -> 1.32.0 x2
[U*]  #115  vscode                         1.105.1 -> 1.106.0
[U.]  #116  vte                            0.80.3 x2 -> 0.82.1 x2
[U*]  #117  xdg-desktop-portal-gnome       48.0 -> 49.0
[U*]  #118  xorg-server                    21.1.18 -> 21.1.20
[U*]  #119  xterm                          402 -> 403
[U*]  #120  yq-go                          4.48.1 -> 4.48.2
[U.]  #121  zenity                         4.1.90 -> 4.2.1
Added packages:
[A.]  #01  fftw-double                        3.3.10
[A.]  #02  gst-plugins-rs                     0.14.2
[A.]  #03  initrd-release                     <none>
[A.]  #04  libglycin                          2.0.5
[A.]  #05  libspelling                        0.4.9
[A.]  #06  openapv                            0.2.0.4 x2
[A+]  #07  papers                             49.1
[A.]  #08  plymouth-initrd-plugins            <none>
[A.]  #09  python3.13-dbus-python             1.4.0
[A.]  #10  python3.13-mako                    1.3.10
[A.]  #11  python3.13-markupsafe              3.0.3
[A.]  #12  python3.13-setuptools              80.9.0
[A+]  #13  showtime                           49.0
[A.]  #14  unit-graphical-session-pre.target  <none>
[A.]  #15  unit-linger-users.service          <none>
[A.]  #16  unit-script-linger-users-start     <none>
[A.]  #17  wait-for-nvidia-devices            <none>
Removed packages:
[R.]  #1  initrd-nixos.conf  <none>
Closure size: 1835 -> 1866 (1783 paths added, 1752 paths removed, delta +31, disk usage +856.1MiB).

[msanft]

On 50a96ed (nixos-unstable), everything works.

Diff from there to the non-working system:

[msanft@bastion:/nix/var/nix/profiles]$ nvd diff system-32-link system-40-link
<<< system-32-link
>>> system-40-link
Version changes:
[D.]  #1  nixos-system-bastion  25.11.20251116.85a6c4a -> 25.11.20251116.50a96ed
Removed packages:
[R.]  #1  wait-for-nvidia-devices  <none>
Closure size: 1866 -> 1865 (30 paths added, 31 paths removed, delta -1, disk usage -1.4KiB).

[fxzzi]

also noticed something weird on nixos-unstable-small

terraria has broken gfx, firefox can't play video, etc

works on nixos-unstable

[vcunat]

At a glance, my tip for the broken rendering would be PR #461866

[fxzzi]

yep, that's probably it. thanks
the above issue was fixed by setting terraria back to OGL anyway

[msanft]

didn't occur anymore for me in nixos-unstable