Skip to content

Java driver on macOS fails with "operation not permitted" #28157

Open
Open
Java driver on macOS fails with "operation not permitted"#28157
Labels
type/bug

Description

@eberleant
eberleant
opened on Jun 22, 2026

Nomad version

Output from nomad version

Nomad v1.10.5
BuildDate 2025-09-09T14:36:45Z
Revision a3b86c697f38ab032e1acaae8503ed10815bc4a2

Also reproduced on v2.0.3. v1.10.4 is working as expected.

Operating system and Environment details

macOS 14.7.4 with Apple M3 Pro chip

We also reproduced this on an Apple M5 Pro (with Nomad v2.0.3; 1.x didn't work on the M5 due to #27799)

Issue

Since Nomad v1.10.5, we are running into a permission issue when using the java driver on macOS (rootless). The job fails to start with the error:

failed to launch command with executor: rpc error: code = Unknown desc = failed to start command path="/usr/bin/java" --- args=["/usr/bin/java" "-jar" "/tmp/JavaExample.jar"]: fork/exec /usr/bin/java: operation not permitted

Using the raw_exec driver instead works.

We aren't running Nomad as root - we weren't sure if that was supposed to be necessary. The documentation states:

On Linux, Nomad must run as root since it will use chroot and cgroups which require root privileges.

but we didn't see anything specific to macOS.

Reproduction steps

  1. Start nomad client using /path/to/nomad agent -dev
  2. http://localhost:4646 -> Run Job
  3. Enter the job file -> Plan -> Run

Expected Result

It should run successfully.

Actual Result

It fails with the error:

failed to launch command with executor: rpc error: code = Unknown desc = failed to start command path="/usr/bin/java" --- args=["/usr/bin/java" "-jar" "/tmp/JavaExample.jar"]: fork/exec /usr/bin/java: operation not permitted

Job file (if appropriate)

job "example-job" {

  type = "batch"

  reschedule {
    attempts = 0
  }

  task "example-task" {

    restart {
      attempts = 0
      mode = "fail"
    }

    driver = "java"
    config {
      jar_path = "/tmp/JavaExample.jar"
    }
  }
}

JavaExample.jar was just a Hello World application.

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

==> No configuration files loaded
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 127.0.0.1:4646; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
            Bind Addrs: HTTP: [127.0.0.1:4646]; RPC: 127.0.0.1:4647; Serf: 127.0.0.1:4648
                Client: true
             Log Level: DEBUG
               Node Id: 07655ba9-65cd-1e54-b424-304af8336117
                Region: global (DC: dc1)
                Server: true
               Version: 1.10.5

==> Nomad agent started! Log data will stream in below:

    2026-06-22T17:02:03.870-0400 [DEBUG] nomad: issuer not set; OIDC Discovery endpoint for workload identities disabled
    2026-06-22T17:02:03.871-0400 [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:96c80d2c-98bb-d6b2-70cd-b899831d7c45 Address:127.0.0.1:4647}]"
    2026-06-22T17:02:03.871-0400 [INFO]  nomad.raft: entering follower state: follower="Node at 127.0.0.1:4647 [Follower]" leader-address= leader-id=
    2026-06-22T17:02:03.871-0400 [INFO]  nomad: serf: EventMemberJoin: DCWT260CD-M.global 127.0.0.1
    2026-06-22T17:02:03.871-0400 [INFO]  nomad: starting scheduling worker(s): num_workers=11 schedulers=["service", "batch", "system", "sysbatch", "_core"]
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=3a287d7a-f6de-193d-9f77-4d64ff2e3e6f index=1 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=21b1a59b-4b2c-bad4-6c57-b42884c1d50c index=2 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=ea86e8f9-6491-a0ce-4492-6d54af57da71 index=3 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=17f35439-0bab-89ec-65d4-cdf98cbf85d3 index=4 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=c3a28d4b-e24e-3b37-87c9-9b7ea7d31517 index=5 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=23477e2a-85ea-2560-71e0-df308cddf732 index=6 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=63ca26d8-3049-db91-764f-922dfabc04db index=7 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=852e515a-f1bc-42ad-04c1-7d8aed446e35 index=8 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=3a287d7a-f6de-193d-9f77-4d64ff2e3e6f
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=21b1a59b-4b2c-bad4-6c57-b42884c1d50c
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=63ca26d8-3049-db91-764f-922dfabc04db
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=23477e2a-85ea-2560-71e0-df308cddf732
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=ea86e8f9-6491-a0ce-4492-6d54af57da71
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=17f35439-0bab-89ec-65d4-cdf98cbf85d3
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=39df8759-75d4-35f9-6b7f-223bcdd0abfd index=9 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=bd299067-dac2-6250-6fee-dfb28791b722 index=10 of=11
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad: started scheduling worker: id=ddcc9b5b-bfa9-5047-61e0-894b300422b6 index=11 of=11
    2026-06-22T17:02:03.871-0400 [INFO]  nomad: started scheduling worker(s): num_workers=11 schedulers=["service", "batch", "system", "sysbatch", "_core"]
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=852e515a-f1bc-42ad-04c1-7d8aed446e35
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=bd299067-dac2-6250-6fee-dfb28791b722
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=39df8759-75d4-35f9-6b7f-223bcdd0abfd
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6
    2026-06-22T17:02:03.871-0400 [DEBUG] nomad.keyring.replicator: starting encryption key replication
    2026-06-22T17:02:03.871-0400 [DEBUG] worker: running: worker_id=c3a28d4b-e24e-3b37-87c9-9b7ea7d31517
    2026-06-22T17:02:03.871-0400 [INFO]  nomad: adding server: server="DCWT260CD-M.global (Addr: 127.0.0.1:4647) (DC: dc1)"
    2026-06-22T17:02:03.871-0400 [DEBUG] agent.plugin_loader.raw_exec.id_validator: user range configured: plugin_dir="" denied range=""
    2026-06-22T17:02:03.871-0400 [DEBUG] agent.plugin_loader.raw_exec.id_validator: group range configured: plugin_dir="" denied range=""
    2026-06-22T17:02:03.871-0400 [DEBUG] agent.plugin_loader.exec.id_validator: user range configured: plugin_dir="" denied range=""
    2026-06-22T17:02:03.871-0400 [DEBUG] agent.plugin_loader.exec.id_validator: group range configured: plugin_dir="" denied range=""
    2026-06-22T17:02:03.872-0400 [DEBUG] agent.plugin_loader.docker: using client connection initialized from environment: plugin_dir=""
    2026-06-22T17:02:03.872-0400 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2026-06-22T17:02:03.872-0400 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2026-06-22T17:02:03.872-0400 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2026-06-22T17:02:03.872-0400 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2026-06-22T17:02:03.872-0400 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2026-06-22T17:02:03.872-0400 [INFO]  client: using state directory: state_dir=/private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3144624240
    2026-06-22T17:02:03.873-0400 [INFO]  client: using alloc directory: alloc_dir=/private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3432027600
    2026-06-22T17:02:03.873-0400 [INFO]  client: using dynamic ports: min=20000 max=32000 reserved=""
    2026-06-22T17:02:03.880-0400 [DEBUG] consul.sync: sync complete: registered_services=3 deregistered_services=0 registered_checks=3 deregistered_checks=0 failures=0
    2026-06-22T17:02:03.901-0400 [DEBUG] client.fingerprint_mgr: built-in fingerprints: fingerprinters=["arch", "cni", "consul", "cpu", "host", "host_volume_plugins", "landlock", "memory", "network", "nomad", "plugins_cni", "signal", "storage", "vault", "env_aws", "env_gce", "env_azure", "env_digitalocean"]
    2026-06-22T17:02:03.901-0400 [DEBUG] client.fingerprint_mgr: CNI config dir is not set or does not exist, skipping: cni_config_dir=/opt/cni/config
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr.cpu: detected CPU model: name="Apple M3 Pro"
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=consul initial_period=15s
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr.cpu: detected CPU efficiency core speed: mhz=2748
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr.cpu: detected CPU performance core speed: mhz=4056
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr.cpu: detected CPU efficiency core count: cores=6
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr.cpu: detected CPU performance core count: cores=5
    2026-06-22T17:02:03.904-0400 [DEBUG] client.fingerprint_mgr.cpu: detected CPU core count: cores=11
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.host_volume_plugins: detected plugin built-in: plugin_id=mkdir version=0.0.1
    2026-06-22T17:02:03.929-0400 [WARN]  client.fingerprint_mgr.landlock: failed to fingerprint kernel landlock feature: error="landlock not supported on this platform"
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected and no speed specified by user, falling back to default speed: interface=lo0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lo0 IP=127.0.0.1
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: detected interface IP: interface=lo0 IP=::1
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=lo0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=gif0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=stf0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=anpi1 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=anpi0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=en4 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=en5 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=en1 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=en2 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=en3 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=bridge0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=ap1 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=en0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=awdl0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=llw0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=utun0 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=utun1 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=utun2 mbits=1000
    2026-06-22T17:02:03.929-0400 [DEBUG] client.fingerprint_mgr.network: link speed could not be detected, falling back to default speed: interface=utun3 mbits=1000
    2026-06-22T17:02:03.929-0400 [WARN]  client.fingerprint_mgr.cni_plugins: failed to read CNI plugins directory: cni_path=/opt/cni/bin error="open /opt/cni/bin: no such file or directory"
    2026-06-22T17:02:03.932-0400 [DEBUG] client.fingerprint_mgr: fingerprinting periodically: fingerprinter=vault initial_period=15s
    2026-06-22T17:02:03.934-0400 [DEBUG] client.fingerprint_mgr.env_aws: error querying AWS IDMS URL, skipping
    2026-06-22T17:02:03.935-0400 [DEBUG] client.fingerprint_mgr.env_gce: could not read value for attribute: attribute=machine-type error="Get \"http://169.254.169.254/computeMetadata/v1/instance/machine-type\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    2026-06-22T17:02:03.935-0400 [DEBUG] client.fingerprint_mgr.env_gce: error querying GCE Metadata URL, skipping
    2026-06-22T17:02:03.936-0400 [DEBUG] client.fingerprint_mgr.env_azure: could not read value for attribute: attribute=compute/azEnvironment error="Get \"http://169.254.169.254/metadata/instance/compute/azEnvironment?api-version=2019-06-04&format=text\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    2026-06-22T17:02:03.937-0400 [DEBUG] client.fingerprint_mgr.env_digitalocean: failed to request metadata: attribute=region error="Get \"http://169.254.169.254/metadata/v1/region\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    2026-06-22T17:02:03.937-0400 [DEBUG] client.fingerprint_mgr: detected fingerprints: node_attrs=["arch", "consul", "cpu", "host", "host_volume_plugins", "network", "nomad", "signal", "storage"]
    2026-06-22T17:02:03.937-0400 [INFO]  client.plugin: starting plugin manager: plugin-type=csi
    2026-06-22T17:02:03.937-0400 [INFO]  client.plugin: starting plugin manager: plugin-type=driver
    2026-06-22T17:02:03.937-0400 [INFO]  client.plugin: starting plugin manager: plugin-type=device
    2026-06-22T17:02:03.937-0400 [DEBUG] client.device_mgr: exiting since there are no device plugins
    2026-06-22T17:02:03.937-0400 [INFO]  client.plugin: starting plugin manager: plugin-type=dynamic_host_volume
    2026-06-22T17:02:03.937-0400 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=driver
    2026-06-22T17:02:03.937-0400 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=device
    2026-06-22T17:02:03.937-0400 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=device
    2026-06-22T17:02:03.937-0400 [DEBUG] client.plugin: waiting on plugin manager initial fingerprint: plugin-type=dynamic_host_volume
    2026-06-22T17:02:03.937-0400 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=dynamic_host_volume
    2026-06-22T17:02:03.937-0400 [DEBUG] client.driver_mgr.raw_exec.id_validator: user range configured: driver=raw_exec denied range=""
    2026-06-22T17:02:03.937-0400 [DEBUG] client.driver_mgr.raw_exec.id_validator: group range configured: driver=raw_exec denied range=""
    2026-06-22T17:02:03.937-0400 [DEBUG] client.driver_mgr.exec.id_validator: user range configured: driver=exec denied range=""
    2026-06-22T17:02:03.937-0400 [DEBUG] client.driver_mgr.exec.id_validator: group range configured: driver=exec denied range=""
    2026-06-22T17:02:03.937-0400 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=exec health=undetected description="exec driver unsupported on client OS"
    2026-06-22T17:02:03.938-0400 [DEBUG] client.driver_mgr.docker: using client connection initialized from environment: driver=docker
    2026-06-22T17:02:03.938-0400 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=raw_exec health=healthy description=Healthy
    2026-06-22T17:02:03.938-0400 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=qemu health=undetected description=""
    2026-06-22T17:02:03.939-0400 [DEBUG] client.server_mgr: new server list: new_servers=[127.0.0.1:4647] old_servers=[]
    2026-06-22T17:02:04.015-0400 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=healthy description=Healthy
    2026-06-22T17:02:04.016-0400 [DEBUG] client.driver_mgr: initial driver fingerprint: driver=java health=healthy description=Healthy
    2026-06-22T17:02:04.016-0400 [DEBUG] client.driver_mgr: detected drivers: drivers="map[healthy:[raw_exec docker java] undetected:[exec qemu]]"
    2026-06-22T17:02:04.016-0400 [DEBUG] client.plugin: finished plugin manager initial fingerprint: plugin-type=driver
    2026-06-22T17:02:04.016-0400 [INFO]  client: started client: node_id=977d775d-79fd-e0e6-ba42-ae1ffaf97e7a
    2026-06-22T17:02:04.016-0400 [DEBUG] http: UI is enabled
    2026-06-22T17:02:04.016-0400 [DEBUG] http: UI is enabled
    2026-06-22T17:02:04.020-0400 [DEBUG] consul.sync: sync complete: registered_services=1 deregistered_services=0 registered_checks=1 deregistered_checks=0 failures=0
    2026-06-22T17:02:05.764-0400 [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
    2026-06-22T17:02:05.764-0400 [INFO]  nomad.raft: entering candidate state: node="Node at 127.0.0.1:4647 [Candidate]" term=2
    2026-06-22T17:02:05.764-0400 [DEBUG] nomad.raft: pre-voting for self: term=2 id=96c80d2c-98bb-d6b2-70cd-b899831d7c45
    2026-06-22T17:02:05.764-0400 [DEBUG] nomad.raft: calculated votes needed: needed=1 term=2
    2026-06-22T17:02:05.764-0400 [DEBUG] nomad.raft: pre-vote received: from=96c80d2c-98bb-d6b2-70cd-b899831d7c45 term=2 tally=0
    2026-06-22T17:02:05.764-0400 [DEBUG] nomad.raft: pre-vote granted: from=96c80d2c-98bb-d6b2-70cd-b899831d7c45 term=2 tally=1
    2026-06-22T17:02:05.764-0400 [INFO]  nomad.raft: pre-vote successful, starting election: term=2 tally=1 refused=0 votesNeeded=1
    2026-06-22T17:02:05.764-0400 [DEBUG] nomad.raft: voting for self: term=2 id=96c80d2c-98bb-d6b2-70cd-b899831d7c45
    2026-06-22T17:02:05.764-0400 [DEBUG] nomad.raft: vote granted: from=96c80d2c-98bb-d6b2-70cd-b899831d7c45 term=2 tally=1
    2026-06-22T17:02:05.764-0400 [INFO]  nomad.raft: election won: term=2 tally=1
    2026-06-22T17:02:05.764-0400 [INFO]  nomad.raft: entering leader state: leader="Node at 127.0.0.1:4647 [Leader]"
    2026-06-22T17:02:05.764-0400 [INFO]  nomad: cluster leadership acquired
    2026-06-22T17:02:05.768-0400 [DEBUG] nomad.autopilot: autopilot is now running
    2026-06-22T17:02:05.768-0400 [DEBUG] nomad.autopilot: state update routine is now running
    2026-06-22T17:02:05.769-0400 [INFO]  nomad.core: established cluster id: cluster_id=a03f955e-10ef-19c3-2c76-fc430bc07e48 create_time=1782162125769198000
    2026-06-22T17:02:05.769-0400 [INFO]  nomad: eval broker status modified: paused=false
    2026-06-22T17:02:05.769-0400 [INFO]  nomad: blocked evals status modified: paused=false
    2026-06-22T17:02:05.785-0400 [INFO]  client: node registration complete
    2026-06-22T17:02:05.785-0400 [DEBUG] client: updated allocations: index=1 total=0 pulled=0 filtered=0
    2026-06-22T17:02:05.785-0400 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=0
    2026-06-22T17:02:05.785-0400 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=0 errors=0
    2026-06-22T17:02:05.785-0400 [DEBUG] client: state updated: node_status=ready
    2026-06-22T17:02:05.863-0400 [INFO]  nomad.keyring: initialized keyring: id=09b0db3c-62e8-57ec-ebc8-b03d777bc67d
    2026-06-22T17:02:06.786-0400 [DEBUG] client: state changed, updating node and re-registering
    2026-06-22T17:02:06.789-0400 [INFO]  client: node registration complete
    2026-06-22T17:02:07.072-0400 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="203.958µs"
    2026-06-22T17:02:09.659-0400 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:50354
    2026-06-22T17:02:10.006-0400 [DEBUG] http: request complete: method=GET path="/v1/jobs/statuses?filter=&namespace=%2A&per_page=25" duration="433.584µs"
    2026-06-22T17:02:10.032-0400 [DEBUG] http: request complete: method=GET path=/v1/namespaces duration="169.583µs"
    2026-06-22T17:02:10.032-0400 [DEBUG] http: request complete: method=GET path=/v1/node/pools duration="302.625µs"
    2026-06-22T17:02:10.850-0400 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="144.875µs"
    2026-06-22T17:02:11.534-0400 [DEBUG] http: request complete: method=GET path=/v1/namespaces duration="777.166µs"
==> Newer Nomad version available: 2.0.3 (currently running: 1.10.5)
    2026-06-22T17:02:17.074-0400 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="134.167µs"
    2026-06-22T17:02:19.661-0400 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:50405
    2026-06-22T17:02:19.946-0400 [DEBUG] http: request complete: method=POST path="/v1/jobs/parse?namespace=*" duration=1.422417ms
    2026-06-22T17:02:19.956-0400 [DEBUG] nomad.job.batch_sched: reconciled current state with desired state: eval_id=e7f8d2ff-66f3-fc1a-53ff-262c54586d89 job_id=example-job namespace=default
  results=
  | Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) (disconnect 0) (reconnect 0)
  | Desired Changes for "example-task": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)
  
    2026-06-22T17:02:19.956-0400 [DEBUG] nomad.job.batch_sched: setting eval status: eval_id=e7f8d2ff-66f3-fc1a-53ff-262c54586d89 job_id=example-job namespace=default status=complete
    2026-06-22T17:02:19.957-0400 [DEBUG] http: request complete: method=POST path=/v1/job/example-job/plan duration=3.265042ms
    2026-06-22T17:02:20.853-0400 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=client duration="188.209µs"
    2026-06-22T17:02:24.925-0400 [DEBUG] http: request complete: method=GET path="/v1/jobs/statuses?filter=&namespace=%2A&per_page=25&index=1" duration=14.885608958s
    2026-06-22T17:02:24.926-0400 [DEBUG] worker: dequeued evaluation: worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6 eval_id=0e535b3a-da47-4278-2215-b14aaaa7625f type=batch namespace=default job_id=example-job node_id="" triggered_by=job-register
    2026-06-22T17:02:24.926-0400 [DEBUG] http: request complete: method=POST path=/v1/jobs duration=3.600791ms
    2026-06-22T17:02:24.926-0400 [DEBUG] worker.batch_sched: reconciled current state with desired state: eval_id=0e535b3a-da47-4278-2215-b14aaaa7625f job_id=example-job namespace=default worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6
  results=
  | Total changes: (place 1) (destructive 0) (inplace 0) (stop 0) (disconnect 0) (reconnect 0)
  | Desired Changes for "example-task": (place 1) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)
  
    2026-06-22T17:02:24.931-0400 [DEBUG] worker: submitted plan for evaluation: worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6 eval_id=0e535b3a-da47-4278-2215-b14aaaa7625f
    2026-06-22T17:02:24.931-0400 [DEBUG] worker.batch_sched: setting eval status: eval_id=0e535b3a-da47-4278-2215-b14aaaa7625f job_id=example-job namespace=default worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6 status=complete
    2026-06-22T17:02:24.932-0400 [DEBUG] worker: updated evaluation: worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6 eval="<Eval \"0e535b3a-da47-4278-2215-b14aaaa7625f\" JobID: \"example-job\" Namespace: \"default\">"
    2026-06-22T17:02:24.932-0400 [DEBUG] worker: ack evaluation: worker_id=ddcc9b5b-bfa9-5047-61e0-894b300422b6 eval_id=0e535b3a-da47-4278-2215-b14aaaa7625f type=batch namespace=default job_id=example-job node_id="" triggered_by=job-register
    2026-06-22T17:02:24.933-0400 [DEBUG] client: updated allocations: index=12 total=1 pulled=1 filtered=0
    2026-06-22T17:02:24.933-0400 [DEBUG] client: allocation updates: added=1 removed=0 updated=0 ignored=0
    2026-06-22T17:02:24.933-0400 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task type=Received msg="Task received by client" failed=false
    2026-06-22T17:02:24.933-0400 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=0 errors=0
    2026-06-22T17:02:24.933-0400 [DEBUG] client.alloc_runner.widmgr: retrieving and renewing workload identities: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 num_identities=0
    2026-06-22T17:02:24.936-0400 [DEBUG] client.alloc_runner.task_runner: lifecycle start condition has been met, proceeding: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task
    2026-06-22T17:02:24.936-0400 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task type="Task Setup" msg="Building Task Directory" failed=false
    2026-06-22T17:02:24.939-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task path=/Users/eberleant/Applications/hashicorp/nomad args=["/Users/eberleant/Applications/hashicorp/nomad", "logmon"]
    2026-06-22T17:02:24.945-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task path=/Users/eberleant/Applications/hashicorp/nomad pid=99276
    2026-06-22T17:02:24.945-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task plugin=/Users/eberleant/Applications/hashicorp/nomad
    2026-06-22T17:02:24.946-0400 [DEBUG] http: request complete: method=GET path=/v1/job/example-job duration=1.366125ms
    2026-06-22T17:02:24.956-0400 [DEBUG] http: request complete: method=GET path=/v1/namespaces duration=2.333208ms
    2026-06-22T17:02:24.987-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task version=2
    2026-06-22T17:02:24.987-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task network=unix @module=logmon address=/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/plugin615362013 timestamp=2026-06-22T17:02:24.987-0400
    2026-06-22T17:02:24.989-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task path=/private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3432027600/8e2237c6-4b07-6fbd-f808-e30871a84eb1/alloc/logs/.example-task.stdout.fifo @module=logmon timestamp=2026-06-22T17:02:24.989-0400
    2026-06-22T17:02:24.990-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task path=/private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3432027600/8e2237c6-4b07-6fbd-f808-e30871a84eb1/alloc/logs/.example-task.stderr.fifo @module=logmon timestamp=2026-06-22T17:02:24.990-0400
    2026-06-22T17:02:24.992-0400 [WARN]  client.alloc_runner.task_runner.task_hook.api: error creating task api socket: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task path=/private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3432027600/8e2237c6-4b07-6fbd-f808-e30871a84eb1/example-task/secrets/api.sock error="listen unix /private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3432027600/8e2237c6-4b07-6fbd-f808-e30871a84eb1/example-task/secrets/api.sock: bind: invalid argument"
    2026-06-22T17:02:24.992-0400 [INFO]  client.driver_mgr.java: starting java task: driver=java driver_cfg="{Class: ClassPath: JarPath:/tmp/JavaExample.jar JvmOpts:[] Args:[] ModePID: ModeIPC: CapAdd:[] CapDrop:[] WorkDir:}" args=["-jar", "/tmp/JavaExample.jar"]
    2026-06-22T17:02:24.992-0400 [DEBUG] client.driver_mgr.java: task capabilities: driver=java capabilities=["CAP_AUDIT_WRITE", "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_KILL", "CAP_MKNOD", "CAP_NET_BIND_SERVICE", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYS_CHROOT"]
    2026-06-22T17:02:24.993-0400 [DEBUG] client.driver_mgr.java.executor: starting plugin: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task path=/Users/eberleant/Applications/hashicorp/nomad args=["/Users/eberleant/Applications/hashicorp/nomad", "executor", "{\"LogFile\":\"/private/var/folders/kw/b10x052n0wjd01hnz1gysv000000gq/T/NomadClient3432027600/8e2237c6-4b07-6fbd-f808-e30871a84eb1/example-task/executor.out\",\"LogLevel\":\"debug\",\"FSIsolation\":false,\"Compute\":{\"tc\":36768,\"nc\":11}}"]
    2026-06-22T17:02:24.994-0400 [DEBUG] client.driver_mgr.java.executor: plugin started: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task path=/Users/eberleant/Applications/hashicorp/nomad pid=99277
    2026-06-22T17:02:24.994-0400 [DEBUG] client.driver_mgr.java.executor: waiting for RPC address: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task plugin=/Users/eberleant/Applications/hashicorp/nomad
    2026-06-22T17:02:25.023-0400 [DEBUG] client.driver_mgr.java.executor: using plugin: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task version=2
    2026-06-22T17:02:25.028-0400 [DEBUG] client.driver_mgr.java.executor.stdio: received EOF, stopping recv loop: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task err="rpc error: code = Unavailable desc = error reading from server: EOF"
    2026-06-22T17:02:25.029-0400 [INFO]  client.driver_mgr.java.executor: plugin process exited: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task plugin=/Users/eberleant/Applications/hashicorp/nomad id=99277
    2026-06-22T17:02:25.029-0400 [DEBUG] client.driver_mgr.java.executor: plugin exited: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 driver=java task_name=example-task
    2026-06-22T17:02:25.029-0400 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task type="Driver Failure" msg="failed to launch command with executor: rpc error: code = Unknown desc = failed to start command path=\"/usr/bin/java\" --- args=[\"/usr/bin/java\" \"-jar\" \"/tmp/JavaExample.jar\"]: fork/exec /usr/bin/java: operation not permitted" failed=false
    2026-06-22T17:02:25.030-0400 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task error="failed to launch command with executor: rpc error: code = Unknown desc = failed to start command path=\"/usr/bin/java\" --- args=[\"/usr/bin/java\" \"-jar\" \"/tmp/JavaExample.jar\"]: fork/exec /usr/bin/java: operation not permitted"
    2026-06-22T17:02:25.030-0400 [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task reason="Policy allows no restarts"
    2026-06-22T17:02:25.030-0400 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task type="Not Restarting" msg="Policy allows no restarts" failed=true
    2026-06-22T17:02:25.030-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.stdio: received EOF, stopping recv loop: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task err="rpc error: code = Unavailable desc = error reading from server: EOF"
    2026-06-22T17:02:25.031-0400 [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task plugin=/Users/eberleant/Applications/hashicorp/nomad id=99276
    2026-06-22T17:02:25.031-0400 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task
    2026-06-22T17:02:25.031-0400 [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1 task=example-task
    2026-06-22T17:02:25.031-0400 [INFO]  client.gc: marking allocation for GC: alloc_id=8e2237c6-4b07-6fbd-f808-e30871a84eb1
    2026-06-22T17:02:25.134-0400 [DEBUG] http: request complete: method=GET path=/v1/job/example-job?index=12 duration=177.324292ms
    2026-06-22T17:02:25.134-0400 [DEBUG] client: updated allocations: index=14 total=1 pulled=0 filtered=1
    2026-06-22T17:02:25.134-0400 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2026-06-22T17:02:25.134-0400 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
    2026-06-22T17:02:27.077-0400 [DEBUG] http: request complete: method=GET path=/v1/agent/health?type=server duration="436.417µs"
    2026-06-22T17:02:29.664-0400 [DEBUG] nomad: memberlist: Stream connection from=127.0.0.1:50442

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bug

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Nomad - Community Issues Triage
    Status
    Needs Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions