Skip to content

feat: expose metrics for stopped/exited containers with exit code and OOM detection #3880

Description

Problem

cAdvisor only exposes metrics for running containers. Once a container stops or exits, all its metrics disappear from Prometheus. There is no native way to:

  • Monitor container state transitions (runningexited)
  • Track exit codes via Prometheus labels
  • Detect OOM kills after the fact (the native container_oom_events_total only works for running containers)
  • Alert on containers that keep crashing and restarting

Proposed solution

A new metric docker_container_info (constant gauge = 1) that queries the Docker/Moby socket for all containers (including stopped/exited), exposing:

docker_container_info{
  container_id="abc123",
  name="my-service",
  image="nginx:latest",
  state="exited",
  exit_code="137 (Killed/OOM)"
} 1

Labels:

  • container_id — full container ID
  • name — container name
  • image — image reference
  • state — Docker state: running, exited, paused, etc.
  • exit_code — human-readable exit code label, with OOM detection via ContainerInspect.State.OOMKilled

Use cases

  • Alerting on crash loops: count_over_time(docker_container_info{state="exited"}[5m]) > 3
  • OOM detection: docker_container_info{exit_code=~"137.*"}
  • Inventory of all containers (not just running ones) for auditing

Implementation notes

This was implemented as an internal patch on a fork of cAdvisor (tested on v0.53.x and v0.57.0). It uses ContainerList(All: true) + ContainerInspect from the Moby API. The collector is registered alongside the existing Prometheus collectors in RegisterPrometheusHandler.

Since v0.56.0 dropped github.com/docker/docker in favour of github.com/moby/moby, a clean upstream implementation would use the moby client directly — possibly integrated into the existing Docker container factory rather than as a standalone handler-level collector.

Happy to submit a PR if the maintainers consider this useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions