Problem
cAdvisor only exposes metrics for running containers. Once a container stops or exits, all its metrics disappear from Prometheus. There is no native way to:
- Monitor container state transitions (
running → exited)
- Track exit codes via Prometheus labels
- Detect OOM kills after the fact (the native
container_oom_events_total only works for running containers)
- Alert on containers that keep crashing and restarting
Proposed solution
A new metric docker_container_info (constant gauge = 1) that queries the Docker/Moby socket for all containers (including stopped/exited), exposing:
docker_container_info{
container_id="abc123",
name="my-service",
image="nginx:latest",
state="exited",
exit_code="137 (Killed/OOM)"
} 1
Labels:
container_id — full container ID
name — container name
image — image reference
state — Docker state: running, exited, paused, etc.
exit_code — human-readable exit code label, with OOM detection via ContainerInspect.State.OOMKilled
Use cases
- Alerting on crash loops:
count_over_time(docker_container_info{state="exited"}[5m]) > 3
- OOM detection:
docker_container_info{exit_code=~"137.*"}
- Inventory of all containers (not just running ones) for auditing
Implementation notes
This was implemented as an internal patch on a fork of cAdvisor (tested on v0.53.x and v0.57.0). It uses ContainerList(All: true) + ContainerInspect from the Moby API. The collector is registered alongside the existing Prometheus collectors in RegisterPrometheusHandler.
Since v0.56.0 dropped github.com/docker/docker in favour of github.com/moby/moby, a clean upstream implementation would use the moby client directly — possibly integrated into the existing Docker container factory rather than as a standalone handler-level collector.
Happy to submit a PR if the maintainers consider this useful.
Problem
cAdvisor only exposes metrics for running containers. Once a container stops or exits, all its metrics disappear from Prometheus. There is no native way to:
running→exited)container_oom_events_totalonly works for running containers)Proposed solution
A new metric
docker_container_info(constant gauge = 1) that queries the Docker/Moby socket for all containers (including stopped/exited), exposing:Labels:
container_id— full container IDname— container nameimage— image referencestate— Docker state:running,exited,paused, etc.exit_code— human-readable exit code label, with OOM detection viaContainerInspect.State.OOMKilledUse cases
count_over_time(docker_container_info{state="exited"}[5m]) > 3docker_container_info{exit_code=~"137.*"}Implementation notes
This was implemented as an internal patch on a fork of cAdvisor (tested on v0.53.x and v0.57.0). It uses
ContainerList(All: true)+ContainerInspectfrom the Moby API. The collector is registered alongside the existing Prometheus collectors inRegisterPrometheusHandler.Since v0.56.0 dropped
github.com/docker/dockerin favour ofgithub.com/moby/moby, a clean upstream implementation would use the moby client directly — possibly integrated into the existing Docker container factory rather than as a standalone handler-level collector.Happy to submit a PR if the maintainers consider this useful.