Client API: Prometheus HTTP SD endpoint for local allocations

[dberkerdem]

Proposal

Add GET /v1/client/allocations/prometheus-sd to the client agent HTTP API, serving the node's running allocations as Prometheus HTTP SD target groups.

One target group per allocated port of every running allocation, labeled with __meta_nomad_* labels (namespace, job, task group, allocation, node, port) plus job/group meta as __meta_nomad_meta_<key>. A ?port=<label> query parameter filters to a single port label (e.g. ?port=metrics). The endpoint would serve local client state only and require node:read.

Implementation sketch: ~200 lines in command/agent/ plus a client method. Test plan: unit tests covering target-group rendering, port filtering, ACL enforcement, and IPv6 host-IP bracketing.

Use-cases

Scraping per-allocation metrics with a node-local collector (e.g. a Prometheus agent per node). Each node's collector discovers its own allocations directly from the local client, so scrape-target discovery fans out to the clients instead of funneling through the servers — no server round-trip, no single discovery bottleneck, and discovery keeps working on a node even when servers are briefly unreachable.

[schmichael]

Thanks for the concise write up and excellent contribution @dberkerdem!

My core concern is: does a Prometheus-specific endpoint belong in Nomad? Arguably metrics?format=prometheus is already a big step in that direction, but supporting multiple response types is less of a maintenance burden than an entirely new API. All arguments - subjective or objective - and prior art welcome!

Would this be able to exist as a standalone service, run as a Nomad system job, using workload identity to get the appropriate permissions? Obviously that's more effort and resources than a builtin API, but I'm curious how the two approaches compare.

The endpoint would serve local client state only and require node:read.

I think <namespace>:read-job would be more appropriate as this iterates allocations and exposes their ports. While distinct from service discovery, it's the same concept and therefore should probably match service discovery's ACL: https://developer.hashicorp.com/nomad/api-docs/services#list-services

Since meta is included I think node:read alone would be a security issue and <namespace>:read-job is required.

plus job/group meta as _meta_nomad_meta

What's the use case for exposing all meta vars to Prometheus? Nomad is not prescriptive as to what users should or should not use meta for, so I'm worried this could expose quite a lot of data for some users.