Urunc containers hung as Temrinating in kubernetes

[cmainas]

When we remove a urunc deployment from a Kuberentes cluster, the pod stays as Terminating forever. The monitor process gets killed, but the urunc shim process stays alive.

Steps to reproduce:

1. Deploy any urunc container in Kubernetes. For example:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx-urunc
  name: nginx-urunc
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-urunc
  template:
    metadata:
      labels:
        run: nginx-urunc
    spec:
      runtimeClassName: urunc
      containers:
      - image: harbor.nbfc.io/nubificus/urunc/busybox-qemu-linux-raw:latest
        imagePullPolicy: IfNotPresent
        name: nginx-urunc
        ports:
        - containerPort: 80
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-urunc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx-urunc
  sessionAffinity: None
  type: ClusterIP

2. Wait till it gets ready
3. After the pod started delete the deployment.
4. Check the pods

$ sudo kubectl get pods 
NAME                                         READY   STATUS        RESTARTS   AGE
nginx-urunc-7597d9699c-2fd8z                 1/1     Terminating   0          10m

Output of ps:

$ ps aux | grep urunc
root     2751118  0.1  0.1 1242096 18488 ?       Sl   11:22   0:00 /usr/local/bin/containerd-shim-urunc-v2 -namespace k8s.io -id 388ba96f10965fd054abb22f69ef94b798c2ef855117deaeb47f70dae5720aa8 -address /run/k3s/containerd/containerd.sock
cmainas  2751351  0.0  0.0   6544  2328 pts/5    S+   11:23   0:00 grep --color=auto urunc

The logs from urunc

,"-m","268M","-L","/usr/share/qemu","-cpu","host","-enable-kvm","-display","none","-vga","none","-serial","stdio","-monitor","null","-smp","1","-kernel","/.boot/kernel","-netdev","tap,id=net0,script=no,downscript=no,ifname=tap0_urunc","-device","virtio-net-pci,netdev=net0,host_mtu=1450,mac=5a:0f:e4:7b:ca:9b","-device","virtio-blk-pci,serial=rootfs,drive=rootfs","-drive","format=raw,if=none,id=rootfs,file=/dev/mapper/containerd-pool-snap-3426","-initrd","/urunit.conf","-no-reboot","-nodefaults","-append","panic=-1 console=ttyS0 root=/dev/vda rw ip=10.42.0.167::10.42.0.1:255.255.255.0:urunc:eth0:off retain_initrd URUNIT_CONFIG=/sys/firmware/initrd init=/urunit -- /bin/sh"],"file":"/home/cmainas/urunc/pkg/unikontainers/unikontainers.go:638","func":"github.com/urunc-dev/urunc/pkg/unikontainers.(*Unikontainer).Exec","level":"debug","msg":"Ready to execve VMM","subsystem":"unikontainers","time":"2026-05-22T11:22:42Z"}
2026-05-22T11:22:44.955167+00:00 nuc8 urunc[2751265]: {"args":["urunc","--root","/run/containerd/runc/k8s.io","--log","/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ec35d91a5ba26159e5a595829c54264af30e241d85aed3cf986e62b7796a55bb/log.json","--log-format","json","kill","ec35d91a5ba26159e5a595829c54264af30e241d85aed3cf986e62b7796a55bb","15"],"command":"KILL","file":"/home/cmainas/urunc/cmd/urunc/kill.go:47","func":"main.init.func3","level":"debug","msg":"urunc INVOKED","time":"2026-05-22T11:22:44Z"}
2026-05-22T11:23:14.972797+00:00 nuc8 urunc[2751339]: {"args":["urunc","--root","/run/containerd/runc/k8s.io","--log","/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ec35d91a5ba26159e5a595829c54264af30e241d85aed3cf986e62b7796a55bb/log.json","--log-format","json","kill","ec35d91a5ba26159e5a595829c54264af30e241d85aed3cf986e62b7796a55bb","9"],"command":"KILL","file":"/home/cmainas/urunc/cmd/urunc/kill.go:47","func":"main.init.func3","level":"debug","msg":"urunc INVOKED","time":"2026-05-22T11:23:14Z"}
2026-05-22T11:23:14.974811+00:00 nuc8 urunc[2751339]: {"file":"/home/cmainas/urunc/cmd/urunc/utils.go:128","func":"main.fatalWithCode","level":"error","msg":"no such process","time":"2026-05-22T11:23:14Z"}

[cmainas]

Form previous working versions, the logs show

2026-05-22T12:01:39.884239+00:00 nuc8 urunc[2753509]: {"args":["urunc","--root","/run/containerd/runc/k8s.io","--log","/run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/2fd53004210ca87e3581519948061c8fd6916d6d7c107eba49db6f5d8d4bd9e9/log.json","--log-format","json","kill","--all","2fd53004210ca87e3581519948061c8fd6916d6d7c107eba49db6f5d8d4bd9e9","9"],"command":"KILL","file":"/home/cmainas/urunc/cmd/urunc/kill.go:54","func":"main.init.func3","level":"debug","msg":"urunc INVOKED","time":"2026-05-22T12:01:39Z"}

where the kill command is called with the --all flag. removed from #612

Bringing back the flag, resolves the issue.

[jiwahn]

How about adding an e2e regression test for pod deletion?

[cmainas]

Yeah, we already have some plans to enhance the k8s testing #214. I hoped the current testing we have would have caught such cases, but it did not.

[jiwahn]

I see. Since the current plan in 214 already includes a pod restart regression test, maybe this case can be covered there as well. Thanks for the context.