nomad alloc signal gets a 500 response when alloc has non-sidecar tasks

[dani]

Nomad version

Nomad v2.0.2
BuildDate 2026-05-22T12:46:01Z
Revision 26ed559e8bc8c1776fab706c17bd6ba259f87ae8

Operating system and Environment details

AlmaLinux 9.8, Nomad running using the pre-built binary

Issue

When trying to send a signal to an alloc (without specifying a task), we get a 500 error response if the alloc has non running tasks. For example, if the alloc has a prestart task set with sidecar = false

When trying to signal such an alloc with eg

nomad alloc signal -s SIGKILL 29d2a9c0-a868-fa53-ee36-141821e0d6e9

We get

Error signalling allocation: Unexpected response code: 500 (rpc error: 1 error occurred:
	* Failed to signal task: wait-for, err: Task not running)

Reproduction steps

Run the provided job file.

Once the job is running, try to SIGKILL (or any other sig) the alloc with

nomad alloc signal -s SIGKILL 29d2a9c0-a868-fa53-ee36-141821e0d6e9

Expected Result

Nomad should ignore non running tasks and not reply to the API call as a 500 error

Actual Result

Nomad tries to send the signal to non running tasks, and reply with a 500 error as the task is not running

Job file (if appropriate)

job "test" {
  datacenters = ["dc1"]

  group "test" {
    task "prestart" {
      driver = "docker"
      lifecycle {
        hook = "prestart"
        sidecar = false
      }
      config {
        image = "alpine:latest"
        command = "ls"
      }
      resources {
        cpu = 10
        memory = 10
      }
    }
    task "main" {
      driver = "docker"
      config {
        image = "alpine:latest"
        command = "sleep"
        args    = ["infinity"]
      }
      resources {
        cpu = 10
        memory = 10
      }
    }
  }
}

[schmichael]

Thanks for the issue with repro @dani!

I'm afraid that ignoring all Task not running errors isn't a good solution as what if someone is sending a SIGHUP intending to cause an alloc to reload configs? I think in that scenario they should be informed that their target is not running and therefore could not be signalled.

However you provide an excellent example of where the existing behavior is nonsensical: of course a user wouldn't expect a prestart non-sidecar task to be running if the main task was. They wouldn't expect the main task to be running if the prestart was, so that case shouldn't error either.

The logic for when to error gets pretty non-trivial I'm afraid: if a signal is sent to an alloc with 2 tasks, one of which is in its backoff period between restarts, should the user be informed of an error?

The signal is still sent to running tasks. Nomad does not short-circuit on errors. So another solution would be to change the API to return a list of tasks and whether they were signaled successfully. When a single task is specified the list would contain just that task. It would technically be a backward compatibility break though since in theory some people may be looking for and handling the existing error message.

---

My preference would be to close this issue because signalling individual tasks seems like a valid workaround. I fully acknowledge it's not the nicest API to deal with, but it's not clear to me there's a solution without risks. I'll leave this open for now in case someone has a strong feeling about the best path forward!